VERICAContinuous VeriﬁcationPlatform

Casey Rosenthal, CEO, Cofounder●Built and Managed the Chaos Engineering
Team at Netﬂix●

Built the Chaos Automation Platform (ChAP)●Deﬁned Chaos Engineering and wrote thebookverica.io

Live demos & Continuous Veriﬁcation use cases

Service Level Objectives - What They Are and How to Use Them●
●Service Level Indicators “a carefully deﬁned quantitative measure of some aspect of the level of service thatis provided.” Examples are response latency, error rate, dropped packets
Service Level Objectives “A target value or range of values for a service level that is measured by an SLI. A
natural structure for SLOs is thus SLI ≤ target.”Excerpts From
Site Reliability EngineeringBeyer et. al., 2006

“How can you set a reasonable target if you don’t yet have any users, and therefore may not knowwhat an acceptable failure percentage might look like for them? The honest answer is: just take an
educated guess!”
Excerpt From
Implementing Service Level Objectives
Hidalgo, 2020

Brian Weyer, Senior Software Engineer - Platformverica.io●Loves building software that solves problems●Enjoys designing and 3D printing useful things

OptimalKubernetes

The Kubernetes team has acorporate standard Kubernetes

The Verica
Solution+ new apps and microservices
being built

Pod Best Practices creates apaved path for all teams to use andlets you scale the institutional
knowledge of your Kubernetes
expert and verify industry bestpractices.

+ new developers added withvarying levels of experience= Inconsistent Kubernetes
clusters across the organization
verica.io

SecureKubernetes

Certs are critical to meetcompliance requirements, but...

The Verica
Solution+ unknown dates/unknown
automation

Expired Certs lets a platform
operator verify cluster certiﬁcatesare up-to-date, indicating the needfor rotation before an issue occurs.

+ application deploys andmodifications= Container security stopped
working and no one knew.
verica.io

SecureKubernetes

An aviation customer has a coreapplication that builds a cache on
deployment in an init container

The Verica
Solution+ cache usage grows overtime

Deployment Pod Recovery Timelets a platform operator know if yourkubernetes application can reach aready state within the speciﬁed time+ cache readiness timeincreases= Pod deploy times increase tounsafe durations
verica.io

Jay Landro, Engineering Manager - Kafka

verica.io●

Passionate about event driven systems●Frontender at heart (CSS is the best)●

Loves to ski (Telemark, XC & Downhill)●

Learns by breaking things

Visible

Kubernetes

A security incident has occured
requiring invalidation of logging API
keys+ Logs are crucial data
points for system health+ Metrics are also used forbusiness objectives= Potentially out ofcompliance with ISO 27001
verica.io

The VericaSolution
Log Delivery can help maintaincompliance by validating that logs
produced within a cluster arecollected by a collector and properly
sent to logging provider within areasonable time period

2 companies have merged andapplications now utilize Kafka as
single source of truth+ Variety of skill levels
among programmers+ Kafka clients integrated in
multiple programming
languages= Potential for loss ofavailability or degradidation
verica.io

The Verica
Solution
Consumer Group Rebalance isdesigned to optimize forperformance & availability bytriggering a common consumer
group rebalancing event andrecording the latencies

Organization migrated tocentralized cloud based Kafka
service (like Confluent Cloud)+ Applications distributed
across multiple regions+ End to end latencies are
difficult to determine for Kafka
based applications= Periods of operation
outside of SLO
verica.io

The Verica
Solution
Client Latency adds latency
between clients and the Kafka
cluster to ensure that messages willcontinue to be consumed within thespeciﬁed consumption SLO

Applications running in US-West-2, Kafka service in US-East-1

verica.ioApplications running in US-West-2, Kafka service in US-West-2

verica.io

Demo by Alex Wise

Alex Wise, Sr. Solutions Architect

verica.io

Visible

Kubernetes

A company used a storage driver tocreate persistent volumes in theon-premises Kubernetes cluster+ the driver was forgotten
during upgrade cycles+ eventually it was too far outof specification to function= the cluster could not create
new volumes and no one knew
verica.io

The VericaSolution
Cluster Functional conﬁrms that aKubernetes Cluster can provide thecore functionality that is required ofit.

Visible

Kubernetes

Build confidence in yourinfrastructure by proactively testing
its capabilities and known modes offailure
Cluster Functional verification will test for:● Are all the components in thekube-system namespace operational?
● Are all nodes in a Ready status● Can every node deploy a pod?
● Can pods communicate with eachother?
● Can nodes mount volumes?
verica.io

The Verica
Solution
Cluster Functional conﬁrms that aKubernetes Cluster can provide thecore functionality that is required ofit.

Visible

Kubernetes

A communications company ran alarge Kubernetes cluster thatspanned a complex network topology+ One of the nodes was in anedge computing datacenter+ overloaded networking gear
meant that node saw triple thelatency of other nodes

verica.io= that latency resulted in slow
feedback loops and poor
operational control

The Verica
Solution
Node Reachability conﬁrms thateach node in the Kubernetes cluster
can communicate with each othernode within a given latency
percentile.

Visible

Kubernetes

Build confidence in yourinfrastructure by proactively testing
its capabilities and known modes offailure
Node Reachability verification will test for:● Can pods on each Kubernetes node
send network packets to the othernodes?
● Is the latency of node-nodecommunication above a given thresholdfor a percentile?

verica.io

The Verica
Solution
Node Reachability conﬁrms thateach node in the Kubernetes cluster
can communicate with each othernode within a given latency
percentile.

Visible

Kubernetes

A software development team used a
Helm chart created by a central
Platform team to deploy applications.

+ they changed the port theirapplication listened on+ they did not know to alsochange the port specified in
the Helm chart.

verica.io= traffic could not reach theapplication and they did notunderstand why

The Verica
Solution
Application Endpoint Reachability
checks Kubernetes objects forcommon misconﬁgurations thatprevent traﬃc from reaching theapplication.

Visible

Kubernetes

Build confidence in yourinfrastructure by proactively testing
its capabilities and known modes offailure
Application Endpoint Reachability verification
will test for:● 6 common misconfigurations of pods
● 6 common misconfigurations of services● 2 misconfigurations of ingresses● An end-to-end test of reachability

verica.ioThe Verica
Solution
Application Endpoint Reachability
checks Kubernetes objects forcommon misconﬁgurations thatprevent traﬃc from reaching theapplication.

Conﬁdence in Infrastructure = Higher Availability●Faster remediation, start debugging at the application level○○○

verica.io

Cluster Functional = Conﬁdence in Cluster Operations
Node Reachability = Conﬁdence in the Network
Application Endpoint Reachability = Conﬁdence in Kubernetes Manifests

“The industry is focused on reactive tactics
like detection, remediation, service
degradation, and disaster recovery, but toidentify vulnerabilities in a complex system
we need a proactive approach.
Verica is that approach. ”
JOHN ALLSPAWCEO of Adaptive Capacity Labs,
Leader in software-based
Resilience Engineering

verica.io

Common pain points across industries

●●●●
●
●
●verica.io

Unexpected outages
Broken SLOs
Misconﬁgurations with outsized impact
Clusters limited by headroom
Unmitigated security vulnerabilities
Conﬁguration drift
and more...

CONTINUOUS
VERIFICATION

verica.io

An experimentation
platform that allows
teams to proactively,safely discoversystem weakness
before they disrupt
business outcomes.

Continuous
Veriﬁcation forKubernetes

Do you know when a component isnearing failure and a risk for breaking
SLOs?
Is your cluster secure and defended
against adversaries?
Do you have conﬁdence in app andcluster upgrades?

verica.io

SecureKubernetes

A large Telco uses a popular
container security product to meetcompliance requirements, but...

The Verica
Solution+ configuration changes
over time

Vulnerable Image Detection
Veriﬁcation lets a platform operator
determine if security measures on
the system are adequately
protecting the cluster.

+ application deploys andmodifications= container security stopped
working and no one knew.
verica.ioGetting value where you need it

Platform validation● Ensure new clusters are fully functional● Conﬁrm deployments meet SLAs
Operational Eﬃciency● Identify conﬁguration drift● Provide more information for troubleshooting● Proactively identify SLO impacts
Platform Observability● SLO Calculation and Accuracy● SLO Attainment

verica.io

Cost optimization● Node instance typeoptimization● Pod resource right-sizing
Harden security● Conﬁrm vulnerable images areblocked● Validate protected components
are unalteredOptimalKubernetes

The Kubernetes team has acorporate standard Kubernetes+ new apps andmicroservices being built+ new developers added withvarying levels of experience= inconsistent Kubernetes
clusters across the organization

verica.io

The Verica
Solution
Pod Best Practice Validation
creates a paved path for all teams touse and lets you scale theinstitutional knowledge of yourKubernetes expert and verify
industry best practices.

>_Kubernetes Cluster

Developer● Management andsetup● Veriﬁcation edit/load● View reports andVisualization

verica.io

APICVP Server●●●●On Prem install
Dynamic instrumentationOrchestration
Veriﬁcation monitoring

Kafka Cluster● Production or test systems● No agents needed● Only instrumented while
Veriﬁcations are running● Blast radius controlled

CONTINUOUS
VERIFICATIONFOR YOUverica.io●The latest
veriﬁcations andreleases of Verica●

Our team ofworld-class Kafkaand Kubernetes
engineers●Roadmap Input

Appendix

Evolution of complex system operations

CICDCV

Continuous Integration
was promoted heavily as
part of XP methodology as
a way to achieve this andis now a common
industry norm.

Continuous Delivery
builds on the success of CIby automating the steps ofpreparing code anddeploying it to anenvironment.

Continuous Veriﬁcation
Like CI/CD, Continuous
Veriﬁcation is born out of aneed to navigate increasingly
complex systems.