Autopsy of an automation disaster
Channel: FOSDEM
Date: 2018-03-12

by Jean-François Gagné

At: FOSDEM 2017

You’ve deployed automation, enabled automatic master failover and tested itmany times: great, you can now sleep at night without being paged by a failingserver. However, when you wake up in the morning, things might not have gonethe way you expect. This talk will be about such a surprise.

Once upon a time, a failure brought down a master. Automation kicked in andfixed things. However, a fancy failure, combined with human errors, with anedge-case recovery, and a lack of oversight in automation, lead to a split-brain. This talk will go into details about the convoluted - but still realworld - sequence of events that lead to this disaster. I will cover what couldhave avoided the split-brain and what could have make things easier to fix it.


Room: H.1309 (Van Rijn)
Scheduled start: 2017-02-04 12:05:00