Project: Sonobuoy (Provisioning / Security & Compliance)
File: 2019-08-07-simplified-results-reporting-with-sonobuoy.md

title: Simplified Results Reporting with Sonobuoy 0.15.1 image: /img/sonobuoy.svg excerpt: Sonobuoy 0.15.1 introduces an entirely new feature that promises to make it easier than ever to see the results of your plug-ins. author_name: John Schnake author_url: https://github.com/johnschnake author_avatar: /img/contributors/john-schnake.png categories: [kubernetes, sonobuoy, conformance] tags: ['Sonobuoy Team', 'John Schnake'] date: 2019-08-07 slug: simplified-results-reporting-with-sonobuoy 0.15.1 introduces an entirely new feature that promises to make it easier than ever to see the results of your plug-ins. The new CLI command, `sonobuoy results`, targets a results tarball and can report on which tests pass, fail, or get skipped. Unlike the `sonobuoy e2e` command, this new command has been generalized to work with different types of plug-ins and can be extended to work with different formats. Because we want to be able to report on the actual results of any plug-in, we need to be able to present those results in a standardized way. To accomplish this, we generated a new, simple data format for capturing results. ```go type Item struct { Name string Status string Metadata map[string]string Items []Item } ``` This format is self-referential so that we can represent the results as a tree, which is a natural structure for multiple relationships like nested subtests and daemonsets running the same set of tests on each node. The tree structure also allows us to bubble up failures so that you can see the most important information (whether the plug-in passed or failed) by looking at a single item. Originally, when Sonobuoy introduced the `sonobuoy e2e` command, it was written specifically for the `e2e` plug-in. This customization meant that the CLI tool would make several assumptions: The plug-in was a job-type plug-in, not a daemonset The results would be in `plugins/e2e/results/junit_01.xml` The data would be in the JUnit XML format As a result of these assumptions, a user or tool would need to know all these details to check the results of a single plug-in. In the new standardized approach, we decided to leverage the aggregator server since it has all the plug-in definitions and is already controlling the results data. By adding a post-processing step after results are gathered, we can create, organize, and summarize the results so that downstream clients can more easily process arbitrary results. To define where results are located and what the format is, we added two fields to the plug-in definition: ```go ResultFormat string ResultFiles []string (optional) ``` The `ResultFormat` field allows the user running the plug-in to specify a post-processing format. Currently, we support two formats: `junit` and `raw`. These options correspond to the output of the two most common plug-ins: the e2e plug-in and the systemd-logs plug-in. Each plug-in format applies its own heuristic to determine which files to target and how to process them. The `junit` format will attempt to parse every `*.xml` file as a junit result file. This way you dont even have to know the name of the file to process the results, and we can support plug-ins which create multiple junit outputs. The `raw` format is appropriate when you dont really want specific processing on the resulting files but want access to them, such as when youre creating logs or reports. The post-processing will walk the files generated by the plug-in and record their locations within the tarball so that tools can look them up more

The `ResultFiles` field enables you to manually specify which files to process as results so that you dont have to rely on our heuristics to find your result files (for example, if you produce junit results but dont give them the `.xml` extension). Weve added the `sonobuoy results` command to analyze these new, processed results. Let's take a look at a few examples. If you want a human-readable report of the results that lists summary data as well as failures, run the following command: ```bash $ sonobuoy results $tarball --plugin e2e ``` Heres an example of what the results might look like: ```bash Plugin: e2e Status: passed Total: 3586 Passed: 1 Failed: 1 Skipped: 3584 Failures: <list of test names> ``` Note: The `--plugin` flag defaults to `e2e`, so you dont need to add it if you are inspecting `e2e` results. To see detailed data about all the results, you can change the default `--mode=report` to `--mode=detailed`. The `detailed` mode will write all the leaf nodes of the tree as JSON objects so that you can pipe them easily into a tool like `jq`: ```bash $ sonobuoy results $outfile --mode=detailed {"name":"[sig-storage] In-tree Volumes...","status":"skipped","meta":{"path":"e2e|junit_01.xml"}} ``` Note: In this format, we added a path to the object in the tree within the `meta` field to clarify exactly where the results are from. If you target a daemonset plug-in, you might want to print only a certain section of the tree instead of all the leaf nodes. This is why we added the `--node` flag: ```bash $ sonobuoy results $outfile --mode=detailed --node=kind-control-plane --plugin systemd-logs kind-control-plane|systemdlogs {"HOSTNAME":"kind-control-plane",...} ... ``` In the example above, you may notice we didnt explicitly dump the results as a JSON object like we did with the `junit` results. This approach lets you access the raw file contents. For clarity, the Sonobuoy CLI is outputting two things: the tree path to the file in question and the files contents. But doesnt the prefix prevent handling the raw file contents easily? Yes it does, so if you want to access just the file data itself, use the `--skip-prefix` flag. For instance, if you want the first log item in the systemd logs for a particular node, run the following command: ```bash $ sonobuoy results $outfile --mode=detailed --node=kind-control-plane --plugin systemd-logs --skip-prefix | head -n1 | jq ``` This approach makes the systemd-logs plug-in much more useful since the post-processing is simplified. You can now quickly scan for errors or keywords by dumping and filtering the results as desired. Lastly, if you really want to take a look at the whole, post-processed results file, you can use the `--mode=dump` flag: ```bash $ sonobuoy results $outfile --mode=dump name: e2e status: passed items: name: junit_01.xml status: passed meta: file: results/global/junit_01.xml items: name: ' ... ``` Every plug-in goes through a post-processing step in which we aggregate results into a file at a known location: `plugins/<name>/sonobuoy_results.yaml` The new command `sonobuoy results` can print out that data for any plug-in. It can report how many tests passed or failed, print detailed information about every test, or be used to access files generated by a plug-in without having to handle the entire tarball. We hope these new features will help you understand and script around your test runs. A big thank you goes out to the Sonobuoy community for your continuous feedback; we had a lot of community contributions for this release and want to give a special thanks to and for their contributions. Join the Sonobuoy community: Get updates on Twitter () Chat with us on Slack ( on Kubernetes) Join the Kubernetes Software Conformance Working Group: