Unlocking Diego: Achieving Distributed Robustness Through Simplicity and Reliability
Channel: Cloud Foundry
Date: 2017-06-23

Unlocking Diego: Achieving Distributed Robustness Through Simplicity and Reliability [A] - Nima Kaviani, IBM & Adrian Zankich, Pivotal    

Cloud Foundry makes extensive use of Consul as a distributed key/value store in order to achieve active-passive high availability for components that are harder to operate in parallel. This is done through implementing a distributed locking mechanism that allows the active component to claim ownership of the lock and take responsibility on execution.

However, Consul’s underlying RAFT algorithm becomes fragile when the entire environment undergoes multiple BOSH re-deployments. This results in an array of different failures, e.g., Consul not being able to elect a leader or mistakenly choosing multiple nodes as leaders. These recurring inconsistencies in Consul can potentially reduce reliability in a distributed system like Cloud Foundry.

In Diego, as one of the primary subsystems of Cloud Foundry, we have taken advantage of Consul’s distributed locking mechanism in several components such as the Diego database (BBS), the auctioneer, and the route emitters. As such, we have spent extensive amount of time to understand how these distributed locks can potentially be avoided or redesigned in order to reduce dependencies to Consul. To this point, two of the Diego components, i.e., the route emitters and the database have adopted alternative solutions to achieve high availability.

In this talk, we will go over lessons learned from using Consul in Cloud Foundry, and how we are trying to find alternative solutions that can help with improving robustness and reliability of Diego components without having to use a complex system like Consul.

Nima Kaviani
IBM
Nima Kaviani is a cloud engineer with IBM and a contributor to CF Diego. Nima holds a PhD in computer science and tweets and blogs about Cloud Foundry, distributed systems, and technology in general.

Adrian Zankich
Adrian is an engineer with Pivotal and has worked on several teams contributing to different components in Cloud Foundry. He has been the anchor and also the PM for the infrastructure team and currently a contributor to BOSH.