Fink on Kubernetes: Efficient Management of Massive Alert Streams for Astronomical Objects Identific
Channel: CNCF [Cloud Native Computing Foundation]
Date: 2024-03-22

Don't miss out! Join us at our next Flagship Conference: KubeCon + CloudNativeCon North America in Salt Lake City from November 12 - 15, 2024. Connect with our current graduated, incubating, and sandbox projects as the community gathers to further the education and advancement of cloud native computing. Learn more at https://kubecon.io

Fink on Kubernetes: Efficient Management of Massive Alert Streams for Astronomical Objects Identific - Fabrice Jammes & Julien Peloton, CNRS; Etienne Fayen, Universite Paris-Saclay

Modern astronomical facilities such as the Rubin Observatory produce real-time streams of millions of alerts each night, distributed by large Apache Kafka clusters. In this context, the role of the Fink broker project is to facilitate the manipulation of these large-scale astronomical alert streams for scientific communities whose main interest is neither computer science nor software engineering, and enable new discoveries to be made every night. In this talk, I will describe the cloud-based infrastructure of Fink, relying on state-of-the-art distributed computing frameworks such as Apache Spark, and the integration with Kubernetes to enable real-time decisions. Finally, I will dive into the details of Fink's complex software stack which is rigorously tested in a self-hosted CI/CD environment on Kubernetes and OpenStack. You will walk away with an understanding of modern challenges in astronomy, and how Kubernetes can help push further the frontiers of science!