Indestructible storage in the cloud with Apache BookKeeper
Channel: The ASF
Date: 2020-10-22

Indestructible storage in the cloud with Apache BookKeeper
Anup Ghatage, Ankit Jain, Charan Reddy Guttapalem, Karan Mehta, Venkateswararao Jujjuri

A presentation from ApacheCon @Home 2020
https://apachecon.com/acah2020/

This talk highlights how Apache software, community, and corporate interaction works well together. The Salesforce team goes over how they have implemented a highly durable and available cloud storage service based on Apache Bookkeeper. Specifically, they speak about their requirements, why they chose Apache BookKeeper and the changes they made in cooperation with the Apache community to make it as cloud-aware. As software is increasingly deployed in public cloud environments, foundational platforms such as Apache BookKeeper must also continuously evolve to effectively work in multi availability zone environments and be designed to work around problems unique to such environments. We at Salesforce added to BookKeeper the ability to function effectively in a Multi-AZ public cloud environment. The first step to this was adding awareness in bookies about their location in the cluster. Which then enabled zone aware placement policies and handling of entire zone failures. They also go over how all of these functions without allowing any downtime to upper-level services. All of these changes go hand in hand with the core tenets of Apache BookKeeper's core quorum based storage principles but rethought to work across availability zones in a cloud-native manner. This talk goes over how we manage various challenges such as the creation of ensembles, placement, and replication of ledgers, tolerance to bookie/zone failures, upgrade scenarios and backward compatibility all the while satisfying the durability guarantees promised by BookKeeper in a public cloud environment.

Anup Ghatage
Anup works on Salesforce's Infrastructure platform. Previously, he has worked on database internals, query processing and storage at SAP, Cisco Systems and other companies for more than 7 years. Anup holds a BS from the University of Pune and an MS from Carnegie Mellon University. Ask him to perform some close-up magic / read your mind for you.
Ankit Jain
Ankit has worked in Salesforce big data infrastructure for the past few years after graduating from Carnegie Mellon University. He is passionate about distributed systems and big data.
Charan Reddy Guttapalem
Charan is a PMTS, working on a highly available and durable Storage layer for Database System at Salesforce. He serves as the committer and PMC member for the Apache Bookkeeper project. Previously he worked on Windows Phone client side features and APIs.
Karan Mehta
Karan has worked in Salesforce big data infrastructure for the past few years after graduating from UC Irvine. Current Apache Phoenix PMC member.
Venkateswararao Jujjuri
Currently leading an effort to build a massively scalable, highly performant distributed storage service at Salesforce. Previously an Architect and member of the IBM Cloud, Open Virtualization. Current Apache Bookkeeper PMC member.