Designing for Failure Fault Injection, Circuit Breakers and Fast Recovery
Channel: FOSDEM
Date: 2020-09-20

by Walter Heck

At: FOSDEM 2020
https://video.fosdem.org/2020/UD2.120/designing_failure.webm

Designing for Failure


While we all work very hard to build high-available, fault-tolerant and
resillient applications and infrastructures the end-goal is currently often
something along the lines of loosly-coupled/microservices with zero-downtime in
mind.
Upgrades are tied to CI/CD pipelines and we should be sipping pina coladas
on the beach. Time to unleash the Chaos Monkey, because that is what Netflix
does, so we should try it as well.


Now, the backend DB failed. The middleware application is returning errors, and
your frontend is showing a fancy 5xx.


While each layer is able to scale independently or fail-over to another region,
even a simple timeout @ the DB can cause a cascading failure.


The application is ❮em❯designed to work❮/em❯, not designed to ❮em❯recover from failure❮/em❯.


Designing for failure applies to both software development and infrastructure
architecture, and I'd like to talk about a couple of points to highlight this
paradigm.


Please note that this talk replaces one entitled "Introduction to Metal³" that was due to have been given by Stephen Benjamin, who has sent his apologies but is now unable to attend.

Room: UD2.120 (Chavanne)
Scheduled start: 2020-02-01 12:00:00