Building data pipelines for Anomaly Detection - DevConf.CZ 2022
Channel: DevConf
Date: 01/02/2022

Speaker: Tuhin Sharma

Cloud-native applications. Multiple Cloud providers. Hybrid Cloud. 1000s of VMs and containers. Complex network policies. Millions of connections and requests in any given time window. This is the typical situation faced by a Security Operations Control (SOC) Analyst every single day. In this talk, the speaker talks about the high-availability and highly scalable data pipelines that he built for the following use cases :

- Denial of Service: A device in the network stops working.
- Data Loss : An example is a rogue agent in the network transmitting IP data outside the network
- Data Corruption : A device starts sending erroneous data.

The above can be solved through anomaly detection models. The main challenge here is the data engineering pipeline. With almost 7 Billion events occurring every day, processing and storing that for further analysis is a significant challenge. The machine learning models (for anomaly detection) has to be updated every few hours and requires the pipeline to create the feature store in a significantly small time window.

The core components of the data engineering pipeline are:
- Apache Zookeeper
- Apache Kafka
- Apache Flink
- Apache Pinot
- Apche Spark
- Apache Superset

The event logs are stored in Pinot through Kafka topic. Pinot supports apache kafka based indexing service for realtime data ingestion. Pinot has primitive capabilities to create sliding time window statistics. More complex real-time statistics are computed using Flink. Apache Flink is a stream-processing engine and provides high throughput and low latency. Spark jobs are used for batch processing. Superset is used as BI tool for realtime visualization.

The speaker talks through the architectural decisions and shows how to build a modern real-time stream processing data engineering pipeline using the above tools.

Outline
    The problem: overview
    Architecture
    Real-Time Processing
    Anomaly Detection
    Visualization
    Demo

Sched: https://sched.co/siJK