Spark structured streaming with HTTP REST endpoints serving JSON data | Towards Data Science

Writing distributed applications could be a time-consuming process. While running simple spark.range( 0, 10 ).reduce( _ + _ ) ( A “Hello World” example of Spark ) code on your local machine is easy enough, it eventually gets complicated as you come across more complex real-world use cases, especially in the Structured Streaming world where you want to do streaming aggregations, join with other streams, or with static datasets.
Main challenge isn’t just setting up Spark or writing code but to test it efficiently on local machine before pushing it to remote environments.
Processing streaming data from Kinesis, Kafka, or S3 on your local machine, as you write code, might not be feasible for a number of reasons: 1 ) you don’t have enough computing power available. 2) You have to process data from the “earliest” offset on your messaging queue but there’s too much data to process. Even if you rate limit, it could take hours to process that data.
How do you develop a production-ready spark application on your local machine in such cases? Regular practice is to develop against fake data. Memory Stream and rate streaming can help you replicate the structured streaming behavior with some fake data. Let's take a look at this unit test:
It simply reads fake data from the JSON file and adds it to a memory stream (Line #18). Memory stream works well when you are writing unit or integration tests but you can’t really add more data to it on an ad-hoc basis and watch your streaming job process it. It makes memory streams somewhat less interactive ( unless you are running it in a REPL environment ).
It could have been great if spark had some sort of more interactive way to send data to a structured streaming pipeline during the development. It could save a lot of time. You can stream read data from sockets but the amount of data you can send as a single payload through TCP socket is limited ( unless you change your OS settings ). This could get complicated.
Yup, this is something spark does not have built-in and the implementation, in fact, is pretty straightforward: Integrate a simple HTTP web server with your spark pipeline. The server can put the payload on MemoryStream that your spark application can read from.
Note that this is for testing and running locally only. Since it uses Memory Stream underneath, it is not fault-tolerant. Refer to the fault-tolerance semantics in structured streaming.
Now, how do we do that? You can create a simple HTTP server in Scala:
The code above creates a simple HTTP server that prints the request payload and always sends { “success" : true } response back to the client. We can stream this payload data into spark application by putting it on a streaming source, a memory stream. So, of course, this server must be started with the spark application.
In order to make it re-usable and to be able to create multiple HTTP streaming sources, we can put that code in a wrapper class:
This local HTTP server created will be terminated with spark application. You can simply start a server and read streaming data from HTTP endpoint using:
scala> val httpDF = new HttpServerStream( port = 9999 ).toDFhttpDF: org.apache.spark.sql.DataFramescala> httpDF.printSchema()root|-- value: string (nullable = true)|-- timestamp: timestamp (nullable = true)scala> httpDF.isStreaming()res1: Boolean = true
The value column of the data frame is stringified JSON payload received on HTTP endpoint. Here’s a full example:
Now, you can use your favorite REST API tool to send data to this HTTP endpoint http://localhost:9999 and watch your application process it.
Similarly, multiple HTTP streams can be created on different ports within the same spark application. Let’s take an example of clickstreams from Stream-stream joins.
val impressions = new HttpServerStream( port = 9997 ).toDF   .withColumn( "i", from_json($"value" ..... )   .select( $"i.impressionAdId", $"i.impressionTime" )   .withWatermark( "impressionTime", "5 minutes" )val clicks = new HttpServerStream( port = 9998 ).toDF   .withColumn( "c", from_json($"value" ..... ) )   .select( $"c.clickAdId", $"c.clickTime" )   .withWatermark( "clickTime", "10 minutes" )val joinedDF = impressions.join(    clicks,   expr("""     clickAdId = impressionAdId AND     clickTime >= impressionTime AND     clickTime <= impressionTime + interval "5 minutes"   """))
GGive it a try and let me know what you think of this. Feel free to suggest improvements or point out caveats. Connect with me on LinkedIn. Cheers!