Add annotations to Grafana with Jenkins
By: Gary Forrest
Tags: Jenkins, Grafana, Events, Monitoring

I have found that recording any change event for an environment and then adding those events to your monitoring dashboards is an invaluable asset. It helps to narrow the scope of an issue and helps speed up troubleshooting. From my experience, most issues arise due to changes to an environment, whether its code deploy, database changes, configuration changes, patching, backups or any other change.

Many people use Jenkins for CICD workflows and in this example we will setup Jenkins to record code deployment to Grafana. Then, we will go over how to display this annotation on a Grafana dashboard.

Create a Grafana Token

Login to Grafana as an admin user and go to the Organization settings and choose API Keys. From within this section create a new API key and save the token. For more information see the Grafana API Token guide.

Add the Grafana API token to Jenkins

Login to your Jenkins instance and go the credentials management (typically http://localhost:8080/credentials/). You want to add a token of type secret.

Jenkins grafana token credential

In this example we gave our credential the ID of grafana_token, which will be used in the Jenkins job.

Make sure your Jenkins server has the WithCredentials Plugin installed and enabled.

Test your Grafana API Token

To test that our Grafana API token works and we can see the event in Grafana, create a sample payload file, test_api.txt from this example.

{

“text”: “Test grafana”

“Test”,

“tags”: [ “deployment”, “env:dev”, “application:test”, “commit:123456789” ]

}

Next, lets use this file to make the following api call, update the grafana URL and your Grafana API token.

Jump to the section on adding annotations to your dashboard to verify your test call worked.

Update your Jenkins deployment job to push the event to Grafana

Now that you have a Grafana API token and it has been added to the Jenkins credential store, lets update a Jenkins job to push an event to Grafana. You can use the following Jenkins method to call the grafana api with the event.

When calling the method we pass in the event type (deploy, release, etc).

annotateEventToGrafana(deploy)

You will need to update the code for the types of tags you want to use and the information to display. In this example it has links back to the github commit, to show the code changes for that commit. It is also has links to the Blue Green deploy.

Another item to point out, if you look at lines 13–20, they do not have indentation. This is due to an bug with Jenkins that if you add indentation it causes your Jenkins job to fail.

Add annotations to a Grafana dashboard

Choose a Grafana dashboard you want to modify and select the edit dashboard and go to the Annotations section. Here you need to add a new annotation and use your tags from the test call or your Jenkins job.

Choose grafana as the Data Source, select Enabled (this allows you to turn the annotations on or off on a dashboard). Next do Filter by: Tags and Match any: and enter the tags from your test or Jenkins job.

Grafana annotations

This images shows that we are adding an annotation to the dashboard and it will show any annotation tagged with release, env:dev, application:carnival.

Finally you can see from the image below how a deploy and subsequent increase in application response time is quick to diagnose when you see the release as an annotation. I like having the commit and other information about the change so you can quickly see which application and change caused the issue.

Deploy event with slow response time

See Adding events to Wavefront for another example.