Processing Engines for Big Data
By: Javier Ramos
Tags: Big Data, Etl, Scala, Apache, Spark

Introduction

This article is based on my previous article “Big Data Pipeline Recipe” where I tried to give a quick overview of all aspects of the Big Data world.

The goal of this article is to focus on the “T” of the a Big Data ETL pipeline reviewing the main frameworks to process large amount of data. The main focus will be the Hadoop ecosystem.

But first, let’s review what this phase is all about. Remember, at this point you have ingested raw data that is ready to be processed.

Big Data Processing Phase

The goal of this phase is to clean, normalize, process and save the data using a single schema. The end result is a trusted data set with a well defined schema. Processing frameworks such Spark are used to process the data in parallel in a cluster of machines.

Generally, you would need to do some kind of processing such as:

Validation : Validate data and quarantine bad data by storing it in a separate storage. Send alerts when a certain threshold is reached based on your data quality requirements.

: Validate data and quarantine bad data by storing it in a separate storage. Send alerts when a certain threshold is reached based on your data quality requirements. Wrangling and Cleansing : Clean your data and store it in another format to be further processed, for example replace inefficient JSON with Avro.

: Clean your data and store it in another format to be further processed, for example replace inefficient JSON with Avro. Normalization and Standardization of values

and of values Rename fields

fields …

Remember, the goal is to create a trusted data set that later can be used for downstream systems. This is a key role of a data engineer. This can be done in a stream or batch fashion.

The pipeline processing can be divided in three phases in case of batch processing:

Pre Processing Phase

If the raw data is not clean or not in the right format, you need to pre process it. This phase includes some basic validation, but the goal is to prepare the data to be efficiently processed for the next stage. In this phase, you should try to flatten the data and save it in a binary format such Avro. This will speed up further processing. The idea is that the next phase will perform row level operations, and nested queries are expensive, so flattening the data now will improve the next phase performance.

Trusted Phase

In this phase, data is validated, cleaned, normalized and transformed to a common schema stored in Hive or in another data store. The goal is to create a trusted common data set understood by the data owners. Typically, a data specification is created and the role of the data engineer is to apply transformations to match the specification which includes validation rules, transformations, standardization of values, renaming fields, etc.

The end result is a data set in Parquet or other columnar format that can be easily queried. It is critical that you choose the right partitions and optimize the data to perform internal queries.

You may want to partially pre-compute some aggregations at this stage to improve query performance.

Reporting Phase

This step is optional but often required. Unfortunately, when using a data lake, a single schema will not serve all use cases; this is one difference between a data warehouse and data lake. Querying HDFS is not as efficient as a database or data warehouse, so further optimizations are required.

In this phase, you may need to denormalize the data to store it using different partitions so it can be queried more efficiently by the different stakeholders. The idea is to create different views optimized for the different downstream systems (data marts).

In this phase you can also compute aggregations if you do not use an OLAP engine. The trusted phase does not know anything about who will query the data, this phase optimizes the data for the consumers. If a client is highly interactive, you may want to introduce a fast storage layer in this phase like a relational database for fast queries. Alternatively you can use OLAP engines.

For streaming the logic is the same but it will run inside a defined DAG in a streaming fashion. Spark allows you to join stream with historical data but it has some limitations. OLAP engines are which are better suited to merge real time with historical data.

Processing Frameworks

Some of the tools you can use for processing are:

This is the most well known framework for batch processing. Part of the Hadoop ecosystem, it is a managed cluster which provides incredible parallelism, monitoring and a great UI. It also supports stream processing (structural streaming). Basically Spark runs MapReduce jobs in memory increasing up to 100x times regular MapReduce performance. It integrates with Hive to support SQL and can be used to create Hive tables, views or to query data. It has lots of integrations, supports many formats and has a huge community. It is supported by all cloud providers. It can run on YARN as part of a Hadoop cluster but also in Kubernetes and other platforms. It has many libraries for specific use cases such SQL or machine learning.

The first engine to unify batch and streaming but heavily focus on streaming. It can be used as a backbone for microservices like Kafka. It can run on YARN as part of a Hadoop cluster but since its inception has been also optimized for other platforms like Kubernetes or Mesos. It is extremely fast and provides real time streaming, making it a better option than Spark for low latency stream processing, especially for stateful streams. It also has libraries for SQL, Machine Learning and much more. It is faster than Spark and a better option for data streaming.

Apache Storm is a free and open source distributed real-time computation system.It focuses on streaming and it is a managed solution part of the Hadoop ecosystem. It is scalable, fault-tolerant, guarantees your data will be processed, and is easy to set up and operate.

Another great stateful stream processing engine. Samza allows you to build stateful applications that process data in real-time from multiple sources including Apache Kafka. Managed solution part of the Hadoop Ecosystem that runs on top of YARN.

Apache Beam it is not an engine itself but a specification of an unified programming model that brings together all the other engines. It provides a programming model that can be used with different languages, so developers do not have to learn new languages when dealing with big data pipelines. Then, it plugs different back ends for the processing step that can run on the cloud or on premises. Beam supports all the engines mentioned before and you can easily switch between them and run them in any platform: cloud, YARN, Mesos, Kubernetes. If you are starting a new project, I really recommend starting with Beam to be sure your data pipeline is future proof.

Conclusion

Processing engines are the most famous tool currently used in the Big Data word. As a big data engineer, you will be working with these engines quite often. It is critical that you understand the distributed nature of these engines and you know how to optimize them, secure them and monitor them.

Remember that there are also OLAP engines that provide a single solution to query large amounts of data without the need to write complex transformations but by loading data in a specific format which makes queries much more performant.

For a new project, I really recommend looking into Apache Beam since it provides an abstraction on top of all the other engines allowing you to change the processing engine without changing the code.

For stream processing, specially stateful stream processing, consider Flink or Samza. For batch, use Spark.