15 Steps to Write an Application Prometheus Exporter in GO
By: Jack Yeh
Tags: Guides And Tutorials, Monitoring, Exporters, Prometheus

Background

Exporters are the heart and soul of Prometheus Monitoring Pipelines. If you run into a situation where an exporter does not exist yet, you are encouraged to write your own. We will cover the steps required to make the exporter. No worries, it is quick.

Goal

Write an exporter in GO.

Exporter will call REST API against an application (Mirth Connect in this example) when scraped.

Exporter will convert the result payload into Metrics.

1. Setup GO and Package Dependencies

Follow the steps on https://golang.org/doc/install

Create a folder “my_first_exporter”

go mod init my_first_exporter

go get github.com/prometheus/client_golang

go get github.com/joho/godotenv --> creates go.mod file

--> Installs dependency into the go.mod file

2. Create Entry-point and Import Dependencies

Create main.go file, and paste in the following:

package main



import (

"github.com/joho/godotenv"

"github.com/prometheus/client_golang/prometheus"

"github.com/prometheus/client_golang/prometheus/promhttp"

)

3. Put in Entry-point function main()

func main() {

}

4. Add prometheus metrics endpoint and listen on the server port

func main() {

http.Handle("/metrics", promhttp.Handler())

log.Fatal(http.ListenAndServe(":9101", nil))

}

5. Explore External Services API with curl

The application I am monitoring is MirthConnect. I will be making two API calls:

a. get channel statistics

b. get channel id and name mapping

curl -k --location --request GET 'https://apihost/api/channels/statistics' \

--user admin:admin





curl -k --location --request GET 'https://apihost/api/channels/idsAndNames' \

--user admin:admin

6. Convert the curl call into go http calls, and handle the result parsing

This is the most difficult step in the entire process if you are new to Go. For my example, my endpoints return XML payload. This means I had to deserialize XML with the “encoding/xml” package.

A successful conversion means my GO program can perform the same API calls as the curl commands. Here are some GO packages to read up about:

“crypto/tls" = specify TLS connection options.

“io/ioutil” = reading the result payload from buffer into strings

“net/http” = create transport and clients

“strconv” = converting string to numbers like floating point/integer

You can iterate through the payload and print out their values with the “log” package.

7. Declare Prometheus metric descriptions

In Prometheus, each metric is made of the following: metric name/metric label values/metric help text/metric type/measurement.

Example:

# HELP promhttp_metric_handler_requests_total Total number of scrapes by HTTP status code.

# TYPE promhttp_metric_handler_requests_total counter

promhttp_metric_handler_requests_total{code=”200"} 1.829159e+06

promhttp_metric_handler_requests_total{code=”500"} 0

promhttp_metric_handler_requests_total{code=”503"} 0

For application scrapers, we will define Prometheus metric descriptions, which includes metric name/metric labels/metric help text.

messagesReceived = prometheus.NewDesc(

prometheus.BuildFQName(namespace, "", "messages_received_total"),

"How many messages have been received (per channel).",

[]string{"channel"}, nil,

)

8. Declare the Prometheus Exporter with interface stubs

Custom exporters requires 4 stubs:

a. A structure with member variables

b. A factory method that returns the structure

c. Describe function

d. Collect function

type Exporter struct {

mirthEndpoint, mirthUsername, mirthPassword string

}



func NewExporter(mirthEndpoint string, mirthUsername string, mirthPassword string) *Exporter {

return &Exporter{

mirthEndpoint: mirthEndpoint,

mirthUsername: mirthUsername,

mirthPassword: mirthPassword,

}

} func (e *Exporter) Describe(ch chan<- *prometheus.Desc) {

} func (e *Exporter) Collect(ch chan<- prometheus.Metric) {

}

9. Inside describe function, send it the metric descriptions from step 7.

func (e *Exporter) Describe(ch chan<- *prometheus.Desc) {

ch <- up

ch <- messagesReceived

ch <- messagesFiltered

ch <- messagesQueued

ch <- messagesSent

ch <- messagesErrored

}

10. Move the logic of api calls from 6 into the collect function

Here we want to take the logic from step 6, and instead of printing things out to the screen, sending it into prometheus.Metric channel:

func (e *Exporter) Collect(ch chan<- prometheus.Metric) {

channelIdNameMap, err := e.LoadChannelIdNameMap()

if err != nil {

ch <- prometheus.MustNewConstMetric(

up, prometheus.GaugeValue, 0,

)

log.Println(err)

return

}

ch <- prometheus.MustNewConstMetric(

up, prometheus.GaugeValue, 1,

)



e.HitMirthRestApisAndUpdateMetrics(channelIdNameMap, ch)

}

As you perform the api call, be sure to send in the measurements using prometheus.MustNewConstMetric(prometheus.Desc, metric type, measurement)

For cases where you need to pass in extra labels, include them at the end of the argument list like this:

channelError, _ := strconv.ParseFloat(channelStatsList.Channels[i].Error, 64) ch <- prometheus.MustNewConstMetric(

messagesErrored, prometheus.GaugeValue, channelError, channelName,

)

11. Declare the exporter in main function, and register it

exporter := NewExporter(mirthEndpoint, mirthUsername, mirthPassword)

prometheus.MustRegister(exporter)

Your exporter is now ready to use!

Every time you hit the metrics route, it will perform the api call, and return the result in Prometheus Text file format.

The rest of the steps deal with tidying things up for easier deployment.

12. Move hard coded api paths into flags

So far, we are hard coding a couple of things like application base url, metric route url, and exporter port. We can make the program more flexible by parsing these values from command line flags:

var (

listenAddress = flag.String("web.listen-address", ":9141",

"Address to listen on for telemetry")

metricsPath = flag.String("web.telemetry-path", "/metrics",

"Path under which to expose metrics")

) func main() {

flag.Parse()

...

http.Handle(*metricsPath, promhttp.Handler())

log.Fatal(http.ListenAndServe(*listenAddress, nil))

}

13. Move credentials into Environment variables

What if application endpoint changes place, or login credentials change? We can load these from environment variables. In this example, I am using the godotenv package to help with storing variable values locally in the same folder:

import (

"os"

) func main() {

err := godotenv.Load()

if err != nil {

log.Println("Error loading .env file, assume env variables are set.")

} mirthEndpoint := os.Getenv("MIRTH_ENDPOINT")

mirthUsername := os.Getenv("MIRTH_USERNAME")

mirthPassword := os.Getenv("MIRTH_PASSWORD")

}

14. Include a Makefile for quick building on different platforms

Makefiles can save you a lot of typing during development. For exporters that needs to build to multiple platforms (testing on windows/mac, running in Linux), you can start with the following:

linux:

GOOS=linux GOARCH=amd64 go build

mac:

GOOS=darwin GOARCH=amd64 go build

Simply invoke make mac or make linux to see different executable file show up.

15. Writing a service file to run this go program as a daemon

Depending on where this exporter will run, you can either write a service file, or a Dockerfile.

A simple Centos 7 Service file looks like the following:

[Unit]

Description=mirth channel exporter

After=network.target

StartLimitIntervalSec=0

[Service]

Type=simple

Restart=always

RestartSec=1

WorkingDirectory=/mirth/mirthconnect

EnvironmentFile=/etc/sysconfig/mirth_channel_exporter

ExecStart=/mirth/mirthconnect/mirth_channel_exporter



[Install]

WantedBy=multi-user.target

That’s all!

Throughout the flow, the only difficult step is 6. If you know what calls are available and how to parse them, the rest come naturally. Come create more exporters and get more metrics flowing today!

Thanks for reading, if you need to grab some application metrics quickly, you can contact us at info@teamzerolabs.com!