Chaos Engineering toContinuous Verification@CaseyRosenthal

CONTINUOUS VERIFICATION

@CaseyRosenthalWhat could go wrong in acomplex system?
@CaseyRosenthal

PUSH

STAGINGLOGGINGBILLINGPUSH

STAGINGLOGGINGBILLINGPUSH
PRODUCTIONLOGGINGBILLINGPUSH

STAGINGLOGGINGBILLINGPUSH
PRODUCTIONLOGGINGBILLINGPUSH

STAGINGLOGGINGBILLINGPUSH
PRODUCTIONLOGGINGBILLINGPUSH

STAGINGLOGGINGBILLINGPUSH
PRODUCTIONLOGGINGBILLING@CaseyRosenthal

PQ

PQ

PQ

PQ

PQ

PQ

PQ

@CaseyRosenthal

AAPIB...
Z...

API
A
...

B
...
Z

...

API
A
...

B
...
Z

...

API
A
...

B
...
Z

...

API
A
...

B
...
Z

...

API
A
...

B
...
Z

...

API
A
...

B
...
Z...

API
A
...
o11yB
...
Z

...

API
A
...
o11y

B
...
Z

...

API
A
...
o11y

B
...
Z

...

API
A
...
o11y

B
...
Z

...

@CaseyRosenthal

All components could be
100% correct,
and yet the system exhibits
undesirable behavior.
@CaseyRosenthal

@CaseyRosenthal

@CaseyRosenthal

About the VOID
The Verica Open Incident Database (VOID) makes public
software-related incident reports available to everyone, raisingawareness and increasing understanding of software-based failuresin order to make the internet a more resilient and safe place.

https://www.thevoid.community

@CaseyRosenthal

@CaseyRosenthal

● Myth 1: Remove the People Who Cause Accidents@CaseyRosenthal

● Myth 1: Remove the People Who Cause Accidents● Myth 2: Document Best Practices and Runbooks@CaseyRosenthal

● Myth 1: Remove the People Who Cause Accidents● Myth 2: Document Best Practices and Runbooks● Myth 3: Defend against Prior Root Causes@CaseyRosenthal

@CaseyRosenthal

● Myth 1: Remove the People Who Cause Accidents● Myth 2: Document Best Practices and Runbooks● Myth 3: Defend against Prior Root Causes@CaseyRosenthal

● Myth 1: Remove the People Who Cause Accidents● Myth 2: Document Best Practices and Runbooks● Myth 3: Defend against Prior Root Causes● Myth 4: Measure Reliability Quantitatively@CaseyRosenthal

Mean Time to Resolution
MTTR

The Distribution Matters

Duration & Severity
THEY’RE NOT RELATED!
We analyzed status page data
across almost 7K incidents from 10different companies.
● Only 2 of them showed very
weak correlations between
duration and severity.
● R = -.18 and -.17, respectively(p < .05)

MTTR Fatal Flaws● Provably wrong statistic● Statistically insigniﬁcant data sample● Duration doesn’t correlate with severity● Measurement errors● Unactionable analysis

MTTR Can’t Tell You● How reliable your software or systems are● How agile/effective your team or organization is● If you’re getting better at responding to incidents● Whether the next one will be longer or shorter● How “bad” any given incident is● Myth 1: Remove the People Who Cause Accidents● Myth 2: Document Best Practices and Runbooks● Myth 3: Defend against Prior Root Causes● Myth 4: Measure Reliability Quantitatively@CaseyRosenthal

● Myth 1: Remove the People Who Cause Accidents● Myth 2: Document Best Practices and Runbooks● Myth 3: Defend against Prior Root Causes● Myth 4: Measure Reliability Quantitatively● Myth 5: Avoid Risk

@CaseyRosenthal

● Myth 1: Remove the People Who Cause Accidents● Myth 2: Document Best Practices and Runbooks● Myth 3: Defend against Prior Root Causes● Myth 4: Measure Reliability Quantitatively● Myth 5: Avoid Risk
● Myth 6: Simplify@CaseyRosenthal

ACCIDENTALESSENTIAL

● Myth 1: Remove the People Who Cause Accidents● Myth 2: Document Best Practices and Runbooks● Myth 3: Defend against Prior Root Causes● Myth 4: Measure Reliability Quantitatively● Myth 5: Avoid Risk
● Myth 6: Simplify● Myth 7: Add Redundancy@CaseyRosenthal● Myth 1: Remove the People Who Cause Accidents● Myth 2: Document Best Practices and Runbooks● Myth 3: Defend against Prior Root Causes● Myth 4: Measure Reliability Quantitatively● Myth 5: Avoid Risk
● Myth 6: Simplify● Myth 7: Add Redundancy@CaseyRosenthal

How do we survive
the undesirable effectsof complex systems?
@CaseyRosenthal

How do we
make systems reliable?
@CaseyRosenthal

@CaseyRosenthal

@CaseyRosenthal

Economic Pillars of Complexity@CaseyRosenthal

Economic Pillars of Complexity@CaseyRosenthal

Economic Pillars of Complexity@CaseyRosenthal

Economic Pillars of Complexity@CaseyRosenthal

Economic Pillars of Complexity@CaseyRosenthal

Software Engineering:
the Reversibility Profession@CaseyRosenthal

“The chief merit of [software engineering] is its technicalefficiency, with a premium placed on precision, speed,expert control, continuity, discretion, and optimal returns oninput.”-Merton@CaseyRosenthal

s/bureaucracy/software engineering/@CaseyRosenthal

“The chief merit of bureaucracy is its technical efficiency,with a premium placed on precision, speed, expert control,continuity, discretion, and optimal returns on input.”-Merton@CaseyRosenthal

Software Engineering:the Bureaucratic Profession@CaseyRosenthal

Software Engineering:
doing it WRONG since 1913
@CaseyRosenthal

@CaseyRosenthal

Evolution ofcomplex system
operations.

VERICA | CONTINUOUS VERIFICATIONCI

One of the most eﬃcient methods for uncovering misalignments
in software is to put the code together and run it. Continuous
Integration was promoted heavily as part of XP methodology as
a way to achieve this and is now a common industry norm.

CDContinuous Delivery builds on the success of CI by automatedthe steps of preparing code and deploying it to an environment.
CD tools allow engineers to choose a build that passed the CI
stage and promote that through the pipeline to run in production.

CVLike CI/CD, Continuous Veriﬁcation is born out of a need tonavigate increasingly complex systems. Modern organizations
can’t validate that the internal machinations of the system work
as intended, so instead they verify that the output of the system
matches expectations.

Continuous Verification is a proactive, experimentation tool for verifying system behavior.
How do other things compare?

VERICA | CONTINUOUS VERIFICATION

Don’t fight complexity.
Navigate it.
@CaseyRosenthal

chaos.community

verica.io@CaseyRosenthal

thevoid.community

● Myth 1: Remove the People Who Cause Accidents● Myth 2: Document Best Practices and Runbooks● Myth 3: Defend against Prior Root Causes● Myth 4: Measure Reliability Quantitatively● Myth 5: Avoid Risk
● Myth 6: Simplify● Myth 7: Add Redundancy@CaseyRosenthal

@caseyrosenthal

@CaseyRosenthal