Logging modernization and cybersecurity at scale with MiNiFi, Kafka and Flink   Pierre Villard
Channel: The ASF
Date: 2023-01-12

Logging modernization and cybersecurity at scale with (Mi)NiFi, Kafka and Flink - Pierre Villard

A presentation from ApacheCon 2022

https://apachecon.com/acna2022/slides/01_PierreVillard_Logging_modernization_cybersecurity.pdf

Efficiently collecting logs across a wide set of heterogeneous devices (network equipments, laptops, servers, etc) over a globally distributed network at scale is hard. This is however a requirement for cybersecurity oriented use cases.

In this talk, we'll be discussing an architecture involving Apache MiNiFi agents, Apache NiFi, Apache Kafka and Apache Flink to efficiently collect logs at scale, processing and normalizing very different log patterns and perform streaming analytics over the streams of data to efficiently monitor the overall network and generate alerts when appropriate.

We'll be demoing how this architecture can be applied to a globally distributed network including on-premises and cloud based deployments.