Building Robust Streaming Data Pipelines with Apache Spark - Zak Hassan, Red Hat
Channel: The Linux Foundation
Date: 2017-09-15

Building Robust Streaming Data Pipelines with Apache Spark - Zak Hassan, Red Hat

There are challenges to architecting a solution that will allow for developers to stream data into Kafka and be able to manage dirty data which is always an issue in ETL pipelines. I'd like to share lessons learned and demonstrate how we can put Apache Kafka, Apache Spark and Apache Camel together to provide developers with a continuous data pipeline for the Spark applications. Without data it is very difficult to take advantage of its full capabilities of Spark. Companies sometimes have their data stored in many different systems and Apache Camel allows developers to Extract, Transform and Load their data to many systems Apache Kafka is one example. Apache Kafka is great for aggregating data in a centralized location and Apache Spark already comes with a built in connector to connect to Kafka. I'll also be explaining lessons learned from running these technologies inside docker.

About Zak Hassan
Zak is a Software Engineer on the Data Analytics Platform Team working on Data Science and Machine Learning on OpenShift. Zak Previously worked as a Software Consultant in financial services and insurance industry building end to end software solutions for enterprise customer. Zak spends his spare time working on OpenSource software and trying to innovate . Zak contributes to open source projects and also posts personal projects on https://github.com/zmhassan .