Spark on Kubernetes in Azure by Gleb Smolnik (Eng)
Channel: Engineering Community
Date: 2020-11-10

This year (2020) a new Spark 3.0 version has been released. It improved different aspects of Spark and Kubernetes integration so that building enterprise-ready solutions using this pair is now comparable to any other Spark implementation.
At the same time Microsoft continues development of cloud-based and fully managed Kubernetes solution called Azure Kubernetes Services (AKS).
Hence, the purpose of this session was to review ability to run Spark on Azure Kubernetes Services. Also I wanted check-out how to integrate such a solution with other Azure services (Azure Files, Azure Data Lake gen2, Azure Data Factory, Azure DevOps) in order to simplify orchestration, code productionalization, dependency & configuration management required to build end-to-end platform. I also mentioned several helpful resources and links which could simplify building "Spark on Kubernetes" as well as provide valuable knowledge on the subj.