Big Data Ingestion Options
By: Javier Ramos
Tags: Hadoop, Big Data, Scala, Spark, Python

Introduction

This article is based on my previous article “Big Data Pipeline Recipe” where I gave a quick overview of all aspects of the Big Data world.

In this article, I will review a bit more in detail the critical data ingestion process and talk about the different options.

This is the first process when building a data pipeline and probably, the most critical one. Careful planning and design is required since this process lays the groundwork for the rest of the data pipeline.

Data Ingestion

The first step is to get the data, the goal of this phase is to get all the data you need and store it in raw format in a single repository. This is usually owned by other teams who push their data into Kafka or a data store.

For simple pipelines with not huge amounts of data, you can build a simple microservices workflow that can ingest, enrich and transform the data in a single pipeline(ingestion + transformation), you may use tools such Apache Airflow to orchestrate the dependencies. In this article, we will focus on big data which needs to be split in several phases.

For Big Data it is recommended that you separate ingestion from processing, massive processing engines that can run in parallel are not great to handle blocking calls, retries, back pressure, etc. So, it is recommended that all the data is saved before you start processing it.

You should enrich your data as part of the ingestion by calling other systems to make sure all the data, including reference data has landed into the lake before processing.

There are two modes of ingestion:

Pull : Pull the data from somewhere like a database, file system, a queue or an API

: Pull the data from somewhere like a database, file system, a queue or an API Push: Applications can also push data into your lake but it is always recommended to have a messaging platform as Kafka in between. A common pattern is Change Data Capture(CDC) which allows us to move data into the lake in real time from databases and other systems.

As we already mentioned, It is extremely common to use Kafka or Pulsar as a mediator for your data ingestion to enable persistence, back pressure, parallelization and monitoring of your ingestion. Then, use Kafka Connect to save the data into your data lake. The idea is that your OLTP systems will publish events to Kafka and then ingest them into your lake. This is the preferred option; if source systems can push data into the data lake directly, go with this approach since you won’t have to manage the dependencies on other systems and teams. In general, dependency management is critical for the ingestion process; you will typically source data from a wide range of system, some new, other legacy; and you will need to manage any change on the data or APIs.

Kafka is a key component for Data Ingestion

Domain Driven Design can be used to manage the dependencies, manage change and set the right responsibilities.

If source systems cannot push data into your data lake, and you need to pull data from other systems. Remember: avoid ingesting data in batch directly through APIs; you may call HTTP end-points for data enrichment but remember that ingesting data from APIs it’s not a good idea in the big data world because it is slow, error prone(network issues, latency…) and can bring down source systems. Although, APIs are great to set domain boundaries in the OLTP world, these boundaries are set by data stores(batch) or topics(real time) in the Big Data world. Of course, it always depends on the size of your data but try to use Kafka or Pulsar when possible and if you do not have any other options; pull small amounts of data in a streaming fashion from the APIs, not in batch. For databases, use tools such Debezium to stream data to Kafka (CDC).

Again, to minimize dependencies, it is always easier if the source system push data to Kafka rather than your team pulling the data since you will be tightly coupled with the other source systems. If this is not possible and you still need to own the ingestion process, we can look at two broad categories for ingestion:

Un-Managed Solutions

These are applications that you develop to ingest data into your data lake; you can run them anywhere, this is a custom solution. This is very common when ingesting data from APIs or other I/O blocking systems that do not have an out of the box solution, or when you are not using the Hadoop ecosystem.

The idea is to use streaming libraries to ingest data from different topics, end-points, queues, or file systems. Because you are developing apps, you have full flexibility. Most libraries provide retries, back pressure, monitoring, batching and much more. This is a code yourself approach, so you will need other tools for orchestration and deployment. You get more control and better performance but more effort involved.

You can have a single monolith or microservices communicating using a service bus or orchestrated using an external tool. Some of the libraries available are Apache Camel or Akka Ecosystem (Akka HTTP + Akka Streams + Akka Cluster + Akka Persistence + Alpakka). You can deploy it as a monolith or as microservices depending on how complex is the ingestion pipeline.

If you use Kafka or Pulsar, you can use them as ingestion orchestration tools to get the data and enrich it. Each stage will move data to a new topic creating a DAG in the infrastructure itself by using topics for dependency management. If you do not have Kafka and you want a more visual workflow you can use Apache Airflow to orchestrate the dependencies and run the DAG. The idea is to have a series of services that ingest and enrich the data and then, store it somewhere. After each step is complete, the next one is executed and coordinated by Airflow. Finally, the data is stored in some kind of storage.

Managed Solutions

In this case you can use tools which are deployed in your cluster and used for ingestion. This is common in the Hadoop ecosystem where you have tools such Sqoop to ingest data from your OLTP databases and Flume to ingest streaming data. These tools provide monitoring, retries, incremental load, compression and much more.

Hadoop Ecosystem with ingestion options at the bottom

NiFi is one of these tools that are difficult to categorize. It is a beast on its own. It can be used for ingestion, orchestration and even simple transformations. So in theory, it could solve simple Big Data problems. It is a managed solution. It has a visual interface where you can just drag and drop components and use them to ingest and enrich data. It has over 300 built in processors which perform many tasks and you can extend it by implementing your own.

NiFi workflow

It has its own architecture, so it does not use any database or HDFS but it has integrations with many tools in the Hadoop Ecosystem. You can call APIs, integrate with Kafka, FTP, many file systems and cloud storage. You can manage the data flow performing routing, filtering and basic ETL. For some use cases, NiFi may be all you need.

However, NiFi cannot scale beyond a certain point, because of the inter node communication more than 10 nodes in the cluster become inefficient. It tends to scale vertically better, but you can reach its limit, especially for complex ETL. However, you can integrate it with tools such Spark to process the data. NiFi is a great tool for ingesting and enriching your data.

Conclusion

Data Ingestion is critical, make sure you analyze the different options and choose the approach that minimizes dependencies. Use Domain Driven Design to manage change and set boundaries. When possible, try to get the data push to your data lake rather than pulling it. In case you need to pull it, use managed solution when possible. If you need to pull data, try to use streaming solutions which provide back pressure, persistence and error handling.

Good luck on your data journey!