- Everyone knows they need a liveness probe
and readiness probe configured
in their Kubernetes manifests.
But what many people don't know
is that there are five attributes
that can actually bite you in the ass
and this happens because when you configure
the liveness and readiness probes,
we usually give it only the mandatory configuration fields,
but we have five other fields
that are the period of seconds
that our application is going to be probed.
The timeoutSeconds,
the successThreshold,
the failureThreshold,
and all of those configurations
that come with default attributes can really hurt us.
And there are many stories in the internet,
including Datadog's, that they actually had an outage
because their timeout period for a probe check
was just one second and it just wasn't enough.
So let's go over the liveness probe
and readiness probe configuration
and the default attributes that you want to set
according to your application's configuration.
What are Liveness and Readiness Probes
So let's talk about liveness probe and readiness probe.
So why do we need them?
So we have a readiness probe
that actually acts as the initial check
to make sure that our process is ready to receive traffic.
So maybe we need to go and pull data from a database
in order to cache it and then serve the traffic
and so on and so on.
So we can use a readiness probe
to know and notify Kubernetes
when it is safe to start sending traffic.
The second one is a liveness probe,
which actually helps Kubernetes
check whether our application is still working correctly.
Now it is very important to not lie to yourself
and configure a liveness probe
that does not really check the entirety of your application.
And I'll give you an example,
things that people do is they go
and just return a response 200 for a health check
while their application actually goes
and talks to database and a queue and a cache,
and they don't test all of those connections.
And at the end of the day, it's just lying to yourself.
It's just setting up a liveness probe
that does not really represent your application.
So how do we actually do this?
We have three options.
Number one is we can set an HTTP check.
Obviously this is for your web apps
and for you to go and check a web application
using the HTTP method.
The second one is a TCP named port.
Let's say you have an application,
maybe it's a DNS server.
Maybe it's an app that just simply works with raw TCP ports.
Then this check is for you.
And the third one is an exec command
and this is what happened to Datadog
because they wanted to check a Redis Cluster,
so they used a Redis PING,
so they just executed the command
and they read the exit command
in order to see whether it is alive or not.
So now that we know what are the checks
that we're going to talk about,
there's one check that we are not going to talk about,
which is the startup check.
Startup probe, it is more intended for legacy workloads
and workloads that take a lot of time to spin up,
and if this is something that is relevant to you,
you can read more about it.
So let's dive deeper into the defaults
that we have for all of those configurations
and what should we avoid when we do that.
Default Values You MUST Know
Okay, so let's talk about the default parameters
and the settings that they provide with themselves.
So let's start with the first one
which is initialDelaySeconds.
The default is set to zero.
So this parameter defines how long should Kubernetes wait
before it starts to probe our workload.
Now this can be problematic
in a case where you start up your application
and it has some sort of a startup sequence.
Maybe pulls data from a database and caches it.
Maybe it waits for other resources to become available.
And if we do not set the initialDelaySeconds
that is enough for application to start,
we can actually encounter problems later on
with other parameters that we're going to talk about
like the failureThreshold and successThreshold
where Kubernetes will deem our application inoperable
even though it might be totally okay,
it just did not have enough time to initially start itself.
The second configuration
that we're going to talk about is the periodSeconds.
This configuration determines how long
should Kubernetes wait in between checks
to see whether our application is alive.
The default settings is 10 seconds
and we should ask ourselves,
what type of application are we running
and is 10 seconds enough?
Is it too much or too low?
I'll give you an example.
If I have a payment application,
and it is very, very important for me
to make sure that it is up and running.
And if my health check does not take a lot of resources,
maybe I want to even
initiate a periodSeconds that is lower than 10.
On the other hand, maybe we have an application
that is very heavy lifting, it does a lot of operations.
And number one, the health check itself is costly.
And number two, maybe it's some batch processing
or something that is, I dunno, log collecting
and that is not so important.
Maybe we want to put it into one minute, two minutes,
up to you.
Next, we have the timeoutSeconds.
And as you remember in the Datadog story,
this is actually the one that bite them in the ass.
And the problem here is that it is default to one second.
And maybe your application,
one second is simply not enough,
and you need to give it three, five, 10 seconds to respond.
And this is up to you for configuration.
And the final two parameters that I want to talk about
is the failureThreshold and the successThreshold.
So this is the amount of times
that Kubernetes will actually try to see
and try again and again,
if our workload is working correctly.
So the default for the failureThreshold is three times.
So it can fail three times until Kubernetes
will deem it unworthy for continuing to serve traffic
and try to restart it.
On the other hand, the successThreshold default is one time.
And now let's take two use cases.
On the first use case, let's use the payment one.
So maybe I have a payment settings and a user comes to pay
and he puts in his credit card,
and he clicks on pay, and he gets an error.
This is a catastrophic error for us
because it really costs us money,
and maybe at this point, our customer might leave us
and never perform the purchase again.
So maybe I want to have a failureThreshold of one,
and a successThreshold of five
to really make sure that my service works correctly.
And maybe the period in seconds that I want to check
is five seconds
because this is really a crucial service for me.
On the other hand, we might have the opposite of that.
Maybe we have some service that collects general telemetry
and we have a client that we control
and the client has retries,
and we have many of those different services,
and we don't really want to waste
a lot of resources on that.
So maybe the failureThreshold might be even five
and maybe the successThreshold might be one
and the periodSeconds that we want to examine it
might be one minute, two minute, three minute
because it is really graceful for us,
and nothing really happens
if it does not respond immediately.
Another one thing to note
is that if our application is a bit flaky,
you know, sometimes there are timeouts,
maybe it's error-prone in terms of networking and so on.
Then again, we maybe want to give it more wiggle room
and increase the failureThreshold,
increase the periodSeconds
and give it some time to actually respond
before we decide to kill it.
So I hope that I've convinced you
Never Use The Default Configurations
to customly define your probes fields
and to make sure that you always
have your custom configuration that fits your application
and to never, ever rely
on the default Kubernetes configuration.
And in order to do so,
you can use our own Datree open-source solution
that scans every Kubernetes manifest file in your cluster
and identifies in which liveness and readiness probes
are you not defining the custom fields.
And this way you can make sure
that you actually configured it correctly.

create a linkedin post for me