How To Create A JSON Data Stream With PySpark & Faker | Towards Data Science

If you are familiar with PySpark and with its Structured Streaming API you know how easy it is to express your streaming job as standard batch job, with the difference that a data stream can be treated as a table that is being continuously appended.
Despite writing a stream processing model in straightforward, finding streaming data sources, could be a challenging task, particularly while testing your application before deploying it or setting your very first streaming job while learning PySpark.
To make your life at work easier and your learning process quicker, in this tutorial I will show you how to produce a mock data stream, serialized in JSON format using Python’s faker and random packages.
Also, you will learn how to save your data stream in a S3 bucket (for free) and then read and aggregate it on the fly using PySpark. All this steps will take just 5 minutes.
In order for this process to be so quick and before you read further, I would recommend you to:
Create a free account on AWS as well as a IAM user. Remember to save both ACCESS KEY ID and SECRET ACCESS KEY for later. Set your IAM user so that it has an AmazonS3FullAccess .Install the boto3 package and create a configuration file to use the two keys generated above, programmatically.Get access to Databricks’ Community Edition, then configure a standard cluster and attach it to the notebook where you will run your PySpark code. This task should be fairly intuitive even if you have no prior experience with the platform.Install the remaining packages, among which faker , random , uuid and json .
Create a free account on AWS as well as a IAM user. Remember to save both ACCESS KEY ID and SECRET ACCESS KEY for later. Set your IAM user so that it has an AmazonS3FullAccess .
Install the boto3 package and create a configuration file to use the two keys generated above, programmatically.
Get access to Databricks’ Community Edition, then configure a standard cluster and attach it to the notebook where you will run your PySpark code. This task should be fairly intuitive even if you have no prior experience with the platform.
Install the remaining packages, among which faker , random , uuid and json .
If it is the first time you create a IAM user in S3, as well as a config file, I suggest to read the first paragraph of this tutorial.
Once you are done with these installations (it will take ~30–45 mins), you are ready to start producing a mock data stream!
The very first step is to create a S3bucket with boto3(in this case data-stream-dump ) using the following command:
Then, I create a data class (if data classes and decorators are a new concept for you have look at this tutorial) named Transaction that is made up of 3 fields:
username is a string and will be randomly generated by calling faker.user_name ;
currency is a string and takes a random value among the ones belonging to the currencies list. I limited the currencies to 3, to make the aggregation performed later in PySpark more evident, but you could also generate random currencies with faker.currency .
amount is also a string and is defined as a random integer in between 100 and 200,000.
I can now use Transaction to define a serialize(self) method (yes, you can define methods as part of data classes) that effectively is used to format these 3 fields as a dictionary:
Now every time I call:
print(Transaction().serialize())
I get back a mock bank transaction created by a fake username in a fake currency and of a fake amount:
{‘username’: ‘jjohnson’, ‘currency’: ‘GBP’, ‘amount’: 102884}
The cool thing about working with data classes and the faker package, is that you can keep increasing or changing the number and type of fields to meet your specific needs.
The cool thing about working with data classes and the faker package, is that you can keep increasing or changing the number and type of fields to meet your specific needs.
I am finally ready to define a Producer() function and pass it to a loop that runs 100 times:
Each time the Producer() function is called, it writes a single transaction in json format to a file (uploaded to S3) that as a name takes the standard root transaction_ plus a uuid code to make it unique.
In this case, the loop will generate 100 files with an interval of 3 seconds in between each file, to simulate a real stream of data, where a streaming application listens to an external micro-service.
The full code until this point is available here. I will compile it very soon, but before that, I need to create a streaming job first.
My streaming job will be written in a DataBrick CE notebook, that looks like the one below:
If you wish for your streaming job to listen to what is happening into a S3 bucket, you will need to “mount” your Amazon S3 bucket as a file system.
Mounting means that you can use Databricks’s platform “to interact with Amazon S3 buckets and perform read/write operations on files and folders”.
There is some documentation out there about how to mount a file system in Databricks, but they make you believe it’s only possible by getting a premium license. It turns out that in this great article Sneha Mehrin found a very easy way to get around that.
Everything you need to do, is running the following code in your notebook (remember to use the 2 keys you saved at the beginning of this tutorial):
If you don’t get any errors, it means your S3 file system has been correctly mounted, however, since your bucket is still empty you won’t see any file by running the following command:
%fs ls /mnt/s3Data/ # --> where /s3Data is the folder name
If you instead, were curious and already run the Producer() function (hopefully with less than 100 iterations :D), you will see this type of file structure:
Finally, it’s all set to to write some code with PySpark!
First, I define a SparkSession (not necessary in Databricks CE, but good practice), then I manually declare the data stream schema( JSONschema ) that has to be provided while reading files with readStream :
Note that the Producer() function has converted dictionaries to JSON files, so that the amount type is now read as LongType() integer in PySpark.
What I am doing below, is to read all the files in the mounted folder that start with the transactions root as a stream, then saving them to a ds DataFrame:
If ds.isStreaming is True it means that the streaming process is active.
Then, I wish to calculate the total amount by currency, therefore I just select these 2 fields from ds and perform a simple aggregation (saving it to trxs )
Eventually, I just need to start writing trxs as a stream and save it in memory (that’s enough for the scope of this tutorial). Since I am performing an aggregation, I have to use complete as an outputMode . Note that the streaming written in memory is named transactions_stream_output:
Now, the first operation I would do is to run the PySpark code that starts reading and writing a stream. This is because it can take up to 30 seconds, so in the meanwhile you can jump on something else.
The second operation I would perform, is to run the first Python script that produces data. When you do that, you will see the numbered iterations appearing on the screen:
Also if you check your S3 UI, you will see files being written in the bucket:
Now go back to the Databricks notebook. Hopefully you should see that the stream is active and that the graphs keep changing as new files are written to S3 and read by the streaming job:
A cool feature in PySpark, is that you can easily query the in-memory table created by the streaming job using SQL. As you can see the the aggregation saved in the table will update every time a new file is written in S3 and processed by the streaming job:
Congratulations! If you got to this point, you managed to build a working data streaming job in PySpark that reads JSON files (including one mock bank transaction each) from a S3 bucket and writes them to an in-memory table.
Now that you know how to build a dynamic data class (using field() and the defaulty_factory parameter) you can extend this concept and create your own data set to be streamed. The limit is just your creativity!
If you found this tutorial useful, please leave some claps or follow me if you wish to read more tutorial on similar topics :D