Back To The Future: Data Engineering Trends 2020 & Beyond
By: Ananth Packkildurai
Tags: Data Engineering, Data Architecture, Serverless, Data Management, Cloud Computing

Data Infrastructure

Managed Data Infrastructure & Serverless Computing

In 2020, We saw the cloud platforms continue adopting the open-source data infrastructure solutions — the adoption growing from AWS’s EMR, Google Cloud data proc, and Azure HDInsight to the recent AWS managed Airflow.

Introducing Amazon Managed Workflows for Apache Airflow (MWAA)

Though opinions differ on cloud platforms packaging the opensource, the cloud-managed infrastructure certainly carry many advantages for the consumers to quickly adopt complex infrastructure and focus on the business problems.

2021 & Beyond

The rise of serverless architecture particularly very interesting trend in data engineering. The article summarizes the serverless data ops trend

Dawn of DataOps: Can We Build a 100% Serverless ETL Following CI/CD Principles?

Google Cloud launched a Google Cloud Workflow as a serverless orchestration engine.

Get to know Workflows, Google Cloud’s serverless orchestration engine

In 2021, It is an exciting space to watch how managed data infrastructure and the rise of serverless computing merge.

Cloud datawarehouse

At the beginning of 2010, tightly coupled computing and storage is a strategy to run large scale data processing engines. 2019 is when the industry finally declared the old way of thinking data processing no longer working and acknowledge the cloud datawarehouse system is the way to go.

Hadoop is Dead. Long live Hadoop.

In 2020, Snowflake’s successful IPO reassured the cloud datawarehouse systems are the future. The S3 strong read-after-write consistency guarantee is a significant step in adopting object storage for the cloud datawarehouse system, if not already.

Amazon S3 Update – Strong Read-After-Write Consistency

2021 & Beyond

The cloud datawarehouse system will continue to dominate and increase the adoption in 2021 and beyond. It will be interesting to watch how the cloud datawarehouse systems are tightly integrating with the data management systems.

Cost Optimization

The cloud datawarehouse systems and the managed data infrastructure adds pressure on optimizing the cost of operating the datawarehouse systems. Netflix writes about cost optimization strategies for its data warehouse system.

Byte Down: Making Netflix’s Data Infrastructure Cost-Effective

At the same time, the GPU accelerated workload can provide a strategic business advantage. Pinterest and NVIDIA shared how Pinterest using GPU acceleration for visual search.

Pinterest Trains Visual Search Faster with Optimized Architecture on NVIDIA GPUs

2021 & Beyond

I added cost optimization as a separate section since cost optimization is often an afterthought. The unpredictability of the object storage engines egress and storage cost, handling cold vs. hot data & the need for specialized hardware for a specific workload will be the norm of 2021 and beyond.

Alluxio is one solution that I am aware of providing tiered data processing capabilities, though not tuned for cost optimization.

Accelerate Spark and Hive Jobs on AWS S3 by 10x with Alluxio as a Tiered Storage Solution