Jenkins Events, Logs, and Metrics | by Shawn Stafford | Towards Data Science

How to collect information about Jenkins for analysis and data visualization
If you support a reasonably large Jenkins instance, or you support a large number of instances, you have probably been faced with a performance problem. Like many Continuous Integration (CI) applications, Jenkins works quite well at a small scale but can degrade significantly without proper care and feeding. This article will present several examples of how to export Jenkins events, logs, and metrics to help identify opportunities for improvement.
The Jenkins Prometheus plugin exposes a Prometheus endpoint in Jenkins that allows Prometheus to collect Jenkins application metrics. The plugin is really just a wrapper around the Metrics plugin to expose JVM metrics through a REST endpoint that returns data in a format which Prometheus can understand. In addition to JVM information, the plugin also exposes information about the job queue, executor counts, and other Jenkins-specific information. The Metrics plugin provides a list of the metrics exposed through the endpoint.
Once the Prometheus plugin has been installed in Jenkins, the data it exposes can be configured through the “Manage Jenkins” page:
The Prometheus plugin is superior to most of the other metrics plugins because it returns more system information than many of the other plugins. If you need to send the data to a different destination, you could use one of the Prometheus exporters. If Prometheus is absolutely not an option, however, there are several alternatives. One alternative is the Jenkins Metrics Graphite Reporting plugin. Data exported by this plugin is far more limited than the Prometheus plugin, but it will allow you to get basic information about executor count and HTTP response statistics.
Unlike the Prometheus pull model, the Graphite plugin will push the data to any server capable of accepting Graphite messages. For example, you could configure the InfluxDB Graphite plugin and send metrics directly to InfluxDB. Or you could configure the Logstash Graphite input plugin and send metrics to any output location supported by Logstash.
If you manage a large number of Jenkins instances, configuring these settings through the UI can be tedious. In such cases, the Jenkins REST API can be used to submit a Groovy script to each instance:
curl -v -d "script=$(cat /tmp/script.groovy)" --user username:ApiToken http://jenkins01.yourcompany.com:8080/scriptText
The Groovy code shown below provides an example of how to configure the Jenkins Metrics Graphite plugin to send data to an external system.
import jenkins.metrics.impl.graphite.GraphiteServer;import jenkins.metrics.impl.graphite.PluginImpl;import jenkins.model.*;import org.codehaus.groovy.runtime.InvokerHelper;// Construct an object to represent the Graphite serverString prefix = "jenkins";String hostname = "graphite.yourcompany.com";int port = 2003;GraphiteServer server = new GraphiteServer(hostname, port, prefix);List<GraphiteServer> servers = new ArrayList<GraphiteServer>();servers.add(server);GraphiteServer.DescriptorImpl descriptor = Jenkins.getInstance().getDescriptorByType(GraphiteServer.DescriptorImpl.class);descriptor.setServers(servers);
The Jenkins Statistics Gatherer plugin can be used to send JSON messages for each event to an external REST endpoint. One application of this would be to send the messages to Elasticsearch for visualization within the Kibana web interface. Jenkins events correspond to actions that occur on the Jenkins master, such as:
Project creation (when a job is created, deleted, or updated)
Job execution (when a build starts and finishes)
Job step execution (when each step in the job starts and finishes)
Job queue (when a job enters or changes state in the job queue)
SCM Checkout (when the job checks out files from source control)
There are many ways to publish events to Elasticsearch. Some popular options include:
Logstash HTTP input plugin -> Logstash Elasticsearch output plugin
FluentD HTTP input plugin -> FluentD Elasticsearch output plugin
Confluent REST Proxy -> Kafka -> Logstash Kafka input plugin -> Logstash Elasticsearch output plugin
For the sake of simplicity, this article will stick with Elasticsearch products and assume the use of Logstash as a means to ingest events into Elasticsearch. Regardless of the solution you choose, the process will essentially be the same.
Once the Statistics Gatherer plugin is installed in Jenkins it can be configured to send messages through the Jenkins UI:
The screenshot above shows how the Statistics Gatherer plugin can be configured to send HTTP messages to a Logstash HTTP input plugin listening at http://logstash.yourcompany.com/.
Important Notes:
The “Enable HTTP publishing?” option must be selected in order for the messages to be sent. This option is only visible once the “Advanced...” button is clicked in this configuration section.
The /jenkins-<msg>/ path at the end is optional, but can be useful to help provide some additional information about which Jenkins event type is being submitted by allowing Logstash filters to be defined on the request_path information.
Builds which publish artifacts can produce unique JSON fields for each Artifact, which can exceed the number of fields allowed for an Elasticsearch index. To avoid this, use a Logstash filter to strip out any unwanted fields:
filter {    mutate {        remove_field => [ "[build][artifacts]" ]    }}
As noted earlier, the Jenkins scripting console or REST endpoint can be used to automate the configuration of the plugin. The contents of the Groovy script would look something like this:
import org.jenkins.plugins.statistics.gatherer.StatisticsConfiguration;import jenkins.model.*;String baseUrl = "http://logstash.yourcompany.com";StatisticsConfiguration descriptor = Jenkins.getInstance()    .getDescriptorByType(StatisticsConfiguration.class);descriptor.setQueueUrl("${baseUrl}/jenkins-queue/");descriptor.setBuildUrl("${baseUrl}/jenkins-build/");descriptor.setProjectUrl("${baseUrl}/jenkins-project/");descriptor.setBuildStepUrl("${baseUrl}/jenkins-step/");descriptor.setScmCheckoutUrl("${baseUrl}/jenkins-scm/");descriptor.setQueueInfo(Boolean.TRUE);descriptor.setBuildInfo(Boolean.TRUE);descriptor.setProjectInfo(Boolean.TRUE);descriptor.setBuildStepInfo(Boolean.TRUE);descriptor.setScmCheckoutInfo(Boolean.TRUE);descriptor.setShouldSendApiHttpRequests(Boolean.TRUE);
At the end of the process, what you should have is a collection of Jenkins event messages in Elasticsearch that can then be used in Kibana visualizations and dashboards to make informed decisions about build performance, failure rates, or a variety of other questions.
The Jenkins master and slave processes generate application logs on the filesystem. These logs contain information about the Jenkins process and can be useful to identify problems that may not be easily identified through the user interface. By sending these logs to Elasticsearch, the information can be indexed and searched for patterns using the Kibana web interface. Each line of the log becomes an JSON record in Elasticsearch.
The easiest way to ship the contents of the application logs to Elasticsearch is to use Filebeat, a log shipper provided by Elastic. Filebeat can be configured to consume any number of logs and ship them to Elasticsearch, Logstash, or several other output channels. I would recommend shipping the logs to Logstash so that the appropriate Logstash filters can be applied to parse the lines into JSON fields. This is particularly useful for HTTP access logs, which use a predictable logging format. For example, the COMBINEDAPACHELOG grok filter in Logstash can be used to parse an access log entry into structured JSON data.
The figures below show the Kibana “Discover” interface, which is useful for searching for log entries. The examples demonstrates how to filter for all Jenkins logs where the hostname matches cdptestjml01 and the log line contains the word docker:
filebeat.host.name: cdptestjml01 AND filebeat.source: jenkins AND message: docker
In the latest Kibana 6.5.x release a “Logs” view has been added which allows you to have a streaming view of the logs similar to the view you might get from running a tail -f command on the command-prompt:
Similar to application logs, each Jenkins job generates a console log. These can also be shipped to Elasticsearch. This can be accomplished using Filebeat on the Jenkins master, or by using the Logstash Jenkins Plugin depending on your needs. Using Filebeat would be similar to what is described above, so for illustrative purposes I’ll cover the Logstash Jenkins plugin here.
Once installed, the plugin must be configured to point to a central server. There are several “indexer types” available to choose from. This example shows the use of the SYSLOG indexer type. This will also require that the Logstash server has a syslog input plugin enabled so that it can receive SYSLOG messages as shown here.
As mentioned earlier, if you are managing multiple Jenkins instances it may be easier to use a Groovy script to configure the plugin:
import jenkins.plugins.logstash.LogstashInstallation;import jenkins.plugins.logstash.LogstashInstallation.Descriptor;import jenkins.plugins.logstash.persistence.LogstashIndexerDao.IndexerType;import jenkins.model.*;Descriptor descriptor = (Descriptor)     Jenkins.getInstance().getDescriptor(LogstashInstallation.class);descriptor.type = IndexerType.SYSLOG;descriptor.host = "logstash.yourcompany.com";descriptor.port = 5045;descriptor.key  = "JenkinsTestEnv";descriptor.save();
The following example shows how to enable the Logstash plugin on an individual job configuration page:
Important Notes:
The Logstash plugin can be enabled globally for all jobs, or on each individual job. It may be preferable to collect logs only for specific jobs due to the volume of data that can be generated (especially if a developer happens to enable debug logging in their build script).
There are many plugins that allow you to send or retrieve data from Jenkins. The first step is to identify what data you need and how you will store it. If you wish to collect time series data from Jenkins at regular intervals, then consider using a time series database and visualize the results with Grafana. Popular database options include:
InfluxDB
Prometheus
TimescaleDB
For other data such as events or logs, the Elastic products are by far the most popular option. The Elasticsearch database is a great solution for storing event messages and Kibana is a user friendly interface for creating dashboards to present the data. Although Elastic has attempted to integrate time series functionality into their product, a dedicated time series database combined with Grafana might be easier to use and manage. Once you have historical data from any or all of these sources, you can begin to make informed and rational decisions about the state of your Jenkins environment.