Concurrent data pipelines in golang | Towards Data Science

Data is a vital piece for all applications. Applications mainly receive data, process it and then create some sort of output out of it. The amount of data that is available nowadays is immense, which brings many challenges when trying to make sense out of it and transforming it into useful information. Besides the usual data system process, we also have to consider the data that is generated by the execution of such systems, like logs and metrics. Everything is data.
Therefore, there’s a growing need for data driven applications. It’s not unusual to have systems only dealing with moving and transforming data. From one place to another or from one structure to another. That is even more common in distributed systems, because each system generates data for their own purpose, structured the way it’s easier for them to process. However, those systems still have to communicate, so the data has to be transformed and moved. Moreover, sometimes the data has to be transferred to a different environment, or to a different platform. Even more trouble!
There are very sophisticated tools to do this job, but very often they are a real pain to configure and maintain, not to mention the costs involved. Thus they are not always the best option. Some times, depending on the use case, you can create your own pipelines to move data with a simpler solution. In this article we are going to see how to build data pipelines using golang.
Pipelines are a really good way to express how we want to deal with data. By definition, pipelines are a sequence of steps or actions that are executed. Usually the output of a previous step is used as the input for the following step. When pipelines are designed, each step doesn’t have to worry about other steps’ work, relying only on the inputs and outputs contracts. This makes the design very robust, as it’s based on contracts and abstractions. Also it’s a very flexible design as you can add more steps as long as you don’t break the contracts. Fun!
There are countless ways of doing it. In this article we will focus on how we can leverage go’s concurrency model, which I wrote about here, to create data pipelines.
As we mentioned earlier, data pipelines are a sequence of steps that are executed using the output of a previous step as the input for the next one. That sound like a really good usage for goroutines and channels. We can create each step as a goroutine, and the communications between them as channels. It means that our data pipeline can execute concurrently, which is a great benefit if used wisely. Let’s jump to an example to make things easier.
In this example we are going to create a simple pipeline for reading some GUID’s from a text file, transform them into the input data structure we are going to use for processing, fetch related data from an external service and finally persist the data in a data storage using a batch operation. Here’s what the high level workflow looks like:
As we can see on this diagram, each operation is going to be created using a goroutine (represented by gophers) and tied together using channels. The whole idea on how to do that is that each function (besides the data generator) will receive its input channel as parameter and return the output channel as the return value of the function. Inside each function we are going to spin a goroutine to actually listen to the input channel and act on each one of the items received, as well as send the result on the output channel.
All channels are unbuffered for simplicity, it means that each one of the GUID’s stored on the guids.txt file will pass trough the pipeline individually. The only batch operation will be at the very end, where we persist them into batches to the data storage. Also for simplicity the error handling has been simplified, bear in mind that for a production pipeline you would want to make it more robust. Alright, enough explanation, let’s get cracking with this example. You can find the entire source code in my github repository.
Here’s what the guids.txt file looks like:
ea464197-d864-4f8a-8a60-2d3a6f3d87bf74c1a7bd-b536-4f9e-be36-5e16c491d833c524c1e9-e473-42eb-b74d-97d82575b720d5003029-7e10-4128-8a18-392e384ef0bdb1fbf405-7b3c-4b82-8b51-4c5c2c0cca56b4f6a28c-86b2-4741-89e5-612451346eb7353629f7-c9ba-4ed1-8e1b-b7694381a36ad3b305cb-eb1b-4f8a-8a23-4741960c26a466c03c1a-45f7-4255-b568-669d9dc553d99272d9f4-da5a-4449-8826-f8529ce4f428
The first piece of our pipeline is to read each line of that file, parse it into auuid.UUID type and send it over on a channel. Here’s how this function can look like:
The important part here is that we are creating the output channel (line 13) and returning it (line 37), as well as using it on the goroutine that actually does the job of reading the file and sending the parsed uuid.UUID values (line 15). That is very powerful, because it allow us to use that channel as input for another function straight away, without worrying on how the internal job is done. Also important is the fact that we are closing the channel once the job is done, this way we can safely use the for range statement on that channel without worrying about infinite loops. Let’s jump on to our second step, preparing the data:
Here the function signature changes slightly. We are receiving the channel we want to use as input. However the internal technique of creating the channel output channel (line 16) and returning it (line 26) is still the same. For this specific step we are only using the uuid.UUID received on the input channel and transforming them into a dummy struct inputData as well as logging what is being processed. This is a very basic example, but it illustrates what you can do for data transforming (a very common step in data pipelines) you only have to change apply the same logic for any struct you want. Let’s check a more in depth example with our third step, data fetching:
This function uses pretty much the same technique as before, we receive an input channel and return an output channel. In this case I decided to emulate the external call for simplicity, so we are actually generating in memory random related values for a given GUID. The interesting thing here is that we are actually using concurrency to call this (fake) external service. This allows the rest of the pipeline to still run while we are waiting for a specific value to return form the external value. We still want to wait for all the calls to the external service to finish before closing our output channel (saying that our job for this piece of the pipeline is done).
To do that I introduced a very common pattern when working with asynchronous calls that need to be synchronised back at some point, using sync.WaitGroup. We wait (synchronise) right before calling the close(oc) on line 29. This way we can make sure that all calls to the external service are finished before closing the channel. It happens because this specific step may run slower than the prior step on the data pipeline, as it depends on external services, so all values may be received from the input channel before we can process all of them, calling the external service.
That was a very interesting step, let’s have a look on the last step of our pipeline, the data store:
In this step we are introducing a different technique, batching. This may be useful for services where calling each operation for each item is not very efficient in terms of performance or cost. So in order to workaround that we can batch some items together. I created a simple batching mechanism based on the amount of data we want to add to the batch. You can use any mechanism you want, I’m sure the logic will be very similar. In this case we are batching 7 items at once, so every time we have 7 items to persist to the database, we call the persisting function (fake) and open the next batch. Before finishing we are checking if there is an open batch and save it in that case. Also we are returning a channel with the information of what items were saved and a timestamp. Now we only need to put this pipeline together:
As you can see we call the pipeline in a reversed order. You can change the parameters to make it more clear if you want. However for this case it is working almost like a decorator pattern. If we start with the generateData() function, that is the one that receives no channels but returns one. So the returned channel from that function is used as input parameter for prepareData. The output channel from that function is used as parameter for the fetchData. The output channel is used finally by saveData as input. The last function also returns a channel, that we are going to range over to print which items have been saved. If we execute this example, we are going to see something like:
./data-pipeline-golang2020/06/10 09:49:50 Data ready for processing: {id:ea464197-d864-4f8a-8a60-2d3a6f3d87bf timestamp:1591778990056111000}2020/06/10 09:49:50 Data ready for processing: {id:74c1a7bd-b536-4f9e-be36-5e16c491d833 timestamp:1591778990056467000}2020/06/10 09:49:50 Data ready for processing: {id:c524c1e9-e473-42eb-b74d-97d82575b720 timestamp:1591778990056587000}2020/06/10 09:49:50 Data ready for processing: {id:d5003029-7e10-4128-8a18-392e384ef0bd timestamp:1591778990056744000}2020/06/10 09:49:50 Data ready for processing: {id:b1fbf405-7b3c-4b82-8b51-4c5c2c0cca56 timestamp:1591778990056773000}2020/06/10 09:49:50 Data ready for processing: {id:b4f6a28c-86b2-4741-89e5-612451346eb7 timestamp:1591778990056803000}2020/06/10 09:49:50 Data ready for processing: {id:353629f7-c9ba-4ed1-8e1b-b7694381a36a timestamp:1591778990056814000}2020/06/10 09:49:50 Data ready for processing: {id:d3b305cb-eb1b-4f8a-8a23-4741960c26a4 timestamp:1591778990056984000}2020/06/10 09:49:50 Data ready for processing: {id:66c03c1a-45f7-4255-b568-669d9dc553d9 timestamp:1591778990057220000}2020/06/10 09:49:50 Items saved: {idsSaved:[66c03c1a-45f7-4255-b568-669d9dc553d9 c524c1e9-e473-42eb-b74d-97d82575b720 353629f7-c9ba-4ed1-8e1b-b7694381a36a b4f6a28c-86b2-4741-89e5-612451346eb7 74c1a7bd-b536-4f9e-be36-5e16c491d833 b1fbf405-7b3c-4b82-8b51-4c5c2c0cca56 ea464197-d864-4f8a-8a60-2d3a6f3d87bf] timestamp:1591778990097935000}2020/06/10 09:49:50 Items saved: {idsSaved:[d3b305cb-eb1b-4f8a-8a23-4741960c26a4 d5003029-7e10-4128-8a18-392e384ef0bd] timestamp:1591778990105146000}
We can see that the prepareData function logged 9 items that were ready for being processed, and the main function logged that 2 batches were actually saved, the first one containing 7 items and the second one containing 2 items.
Creating data pipelines is a very common task for engineers dealing with data driven or data intensive applications. Data pipelines are essential to transform and move data from one system to another, one platform to another or even one environment to another. If we understand golang’s concurrency model we can leverage that to create concurrent data pipelines in a very clean and easy way. The main pattern used here is to create and return a channel for each step, this way we can easily pipe all steps together. Using this paradigm we can boost our pipeline with all sort of cools things, like parallel execution for steps, multiple workers for a specific pipeline step, so on.
This technique is not always the right choice, and there are many tools available out there that can be an easier solution depending on each case.