Custom health checks with ArgoCD
By: Mats Iremark
Tags: Argo, Continuous Delivery, Helm, Flux, Kubernetes

ArgoCD

We were doing Helm releases using ArgoCD in combination with Flux Helm Operator. The ability to be able to get Helm values from secrets plus being able to programmatically generate Helm releases using Jsonnet made this combination really suitable for a pure GitOps workflow. It did however have one drawback. The HelmRelease resource could be deployed to the cluster and show up as green in ArgoCD despite the actual upgrade of the Helm release not succeeding.

In this example we are upgrading Chaoskube Helm chart from version 3.1.2 to 3.1.4 (which does not exist). There is no problem to apply the manifest via ArgoCD but the Flux Helm Operator is of course unable to actually download and install this chart. I want ArgoCD to show this as a warning!

NAME RELEASE STATUS MESSAGE AGE

chaoskube chaoskube DEPLOYED helm upgrade succeeded 46h

conditions:

- lastTransitionTime: "2019-12-02T22:11:19Z"

lastUpdateTime: "2019-12-02T22:15:34Z"

message: helm upgrade succeeded

reason: HelmSuccess

status: "True"

type: Released

- lastTransitionTime: "2019-12-04T20:30:44Z"

lastUpdateTime: "2019-12-04T20:34:06Z"

message: 'chart download failed: chart "chaoskube" version "3.1.4" not found in



reason: RepoFetchFailed

status: "False"

type: ChartFetched status:conditions:- lastTransitionTime: "2019-12-02T22:11:19Z"lastUpdateTime: "2019-12-02T22:15:34Z"message: helm upgrade succeededreason: HelmSuccessstatus: "True"type: Released- lastTransitionTime: "2019-12-04T20:30:44Z"lastUpdateTime: "2019-12-04T20:34:06Z"message: 'chart download failed: chart "chaoskube" version "3.1.4" not found in https://kubernetes-charts.storage.googleapis.com/ repository'reason: RepoFetchFailedstatus: "False"type: ChartFetched

Chaoskube 3.1.2 is running fine but not upgraded to 3.1.4

Digging into details in ArgoCD we find out why

A few versions ago Flux Helm Operator started to include status and conditions. This allows us to use the Custom Health Check feature of ArgoCD to provide a status for this type of resource. Lets dig into the documentation at https://argoproj.github.io/argo-cd/operator-manual/health/.

I had never written a single line of Lua code but it wasn’t that hard to figure out what to do. The first iteration of this custom health check looks like this.

helm.fluxcd.io/HelmRelease:

health.lua: |

hs = {}

if obj.status ~= nil then

if obj.status.conditions ~= nil then

for i, condition in ipairs(obj.status.conditions) do

if condition.type == "ChartFetched" and condition.status == "False" then

hs.status = "Degraded"

hs.message = condition.message

end

if condition.type == "Released" and condition.status == "False" then

hs.status = "Degraded"

hs.message = condition.message

end

if condition.type == "Released" and condition.status == "True" then

hs.status = "Healthy"

hs.message = condition.message

end

end

return hs

end

end hs.status = "Progressing"

hs.message = "Waiting for helm release to be installed"

return hs

Now the Lua script will check the conditions for the HelmRelease resource and indicate that it’s Degraded.

Chaoskube HelmRelease now degraded

With a good error message

To be honest this scenario doesn’t really fit being either Degraded or Out of Sync in ArgoCD. But for us, it allowed us to continue working pure GitOps and get feedback on if there were any issues for the Flux Helm Operator to upgrade a Helm Release.