Spinnaker’s Extensibility Reaches New Heights with Plugins
By: Rosalind Benoit
Tags: DevOps, Continuous Delivery, Spinnaker, Software Development, Site Reliability Engineer

Read this post by Ryan Pei, originally published on the Armory blog, for a guide to Spinnaker plugins and all the resources available to plugin developers. Learn how plugins enable maintainable custom stages, resource handlers, cloud provider and SDLC tool integrations, and custom dashboards for Spinnaker.

Before plugins existed…

Just last year, anyone who wanted to extend or customize Spinnaker faced a dilemma — go with something straightforward but a little hacky, or invest significant time and effort. Some “quick but hacky” methods may be familiar to you: SpEL expressions, custom run jobs, webhooks… these all work well in a pinch. But these are limited in functional scope, re-usability, and range of compatibility with different tools. In terms of more robust means of extending Spinnaker, you could either create extensions via classpath injection or fork Spinnaker services. If others wanted your extension, you might be able to add it directly to the open-source Spinnaker project. But these options require in-depth knowledge of Spinnaker’s codebase, so few teams outside the core community made these more heavyweight extensions.

This situation left us at Armory in a bind; we like to experiment and build cool new things with Spinnaker, but each small experiment was burdensome. Furthermore, the community was frustrated with the growing size and scope of each Spinnaker service. As specialized features piled onto services like Orca and Clouddriver, these grew from humble microservices into monoliths. The community wanted to bring the Spinnaker project back to a “lean core” model while still welcoming innovative contributors.

So, to meet this challenge…

Enter stage left StageDefinitionBuilder - the plugin framework

Our team thought about new ways to extend Spinnaker to achieve the happy medium between “quick but hacky” and “robust but rigorous.” We decided to introduce a plugin framework. Since Spinnaker is primarily comprised of Java microservices, we opted for a well-established plugin framework for Java applications, PF4J. We added extension points throughout the project to form the main integration points of this framework. Then, by using these extension points (or not using them at all, a bit more on this later), anyone can make their own extensions via a plugin.

I introduced a lot of vocab here, so allow me to define these words:

Plugin — In the context of Spinnaker, a plugin is an extension adding new functionality or updating existing functionality onto a Spinnaker service. A plugin is added via Halyard (and soon, Kleat) via a plugin configuration. In terms of what should be a plugin versus what shouldn’t, this depends on several factors. If you want a reusable extension that requires code changes in Spinnaker, you don’t want to fork Spinnaker, and you don’t think your extension belongs in the core open source project, then you likely want a plugin. Some examples of Spinnaker plugins can be found here.

Extension point — Plugins need a way to extend Spinnaker functionality. An extension point is a designated interface or abstract class in the Spinnaker project intended explicitly for plugins to extend. Interface and abstract classes in Java are already extendable, so these extension points are like an API contract for plugin developers. An example of an extension point is StageDefinitionBuilder , which can be extended by plugins known colloquially as “custom stages”. These custom stage, when added, appear to a user like any other "default" stage. Plugins using this extension point comprise these custom stages. Note that a plugin does not require an extension point. A plugin can extend any interface or abstract class. There’s also a way to create plugins using Spring components, but we won’t get into detail on that yet.

Plugin framework — The framework includes these extension points and the PF4J libraries that enable these plugins in Spinnaker.

Note that plugins can also span across several Spinnaker services, not just one. This image below depicts two examples of plugins; one is an EventListener -based plugin ( EventListener being an extension point in Echo) and the other is a custom stage plugin which spans several services. This custom stage plugin may extend StageDefinitionBuilder in Orca as well as interfaces in Clouddriver, while adding custom stage UI components in Deck. Both of these plugins change or add to the functionality of these pre-existing services.

So now that you know what plugins are, we’ll discuss what kinds of nifty functionality these plugins can add.

So what do I use this plugin framework for?

Possibilities are endless, but we foresee five categories of extensions for which plugins should be the best way forward:

Custom stages

Custom resources / resource handlers

Cloud providers (target platforms for app deploys)

Integrations with other DevOps / SDLC tools

Custom views/dashboards

Custom stages

These stages already exist in other, non-plugin forms, such as the custom run job and the custom webhook stage. But none of these enable creating any kind of stage you want (while the custom run job is popular, it is limited to implementation via a K8s job, doesn’t enable several desired stage task features, and doesn’t allow much UI customization in Deck).

The Pulumi Stage enabled by the Pulumi Plugin

When creating a new stage, first familiarize yourself with custom run jobs and custom webhooks in case these methods are sufficient for you. Otherwise, you can create a custom stage plugin. Some examples:

pf4jStage (Random Wait stage)

extends the SimpleStage extension point (the "simpler" custom stage extension point in Orca)

extension point (the "simpler" custom stage extension point in Orca) fairly straightforward, this stage adds a random wait time

provides a sense of how the front-end of the stage is created and added to the Deck stage registry; check out this file which shows you how this plugin defines a new stage and new components in Deck’s registries

winstonStage

extends StageDefinitionBuilder , shows the full range of the Orca stage API

, shows the full range of the Orca stage API note this stage only works in a Netflix environment

enables sophisticated custom stages with more options than SimpleStage , such as enabling multiple tasks in each stage, allowing these tasks to be re-run, and configuring other stages to be run before, after, or based on the result of this stage

fetchArtifact

extends SimpleStage

illustrates how to fetch an artifact from an external source that can be used for any stage executions, such as a CLI package

Custom resources and resource handlers

Resources, such as infrastructure resources or pipeline dependencies, cover essentially everything Spinnaker manages and automates. Plugins can handle your own custom resources:

Custom Kubernetes CRD handlers for Clouddriver

these custom handlers enable you to execute custom code when evaluating a manifest’s stability

there are not yet any official extension points, but at least these Spring-based plugins enable you to add your own CRD handlers without having to fork your own release of Clouddriver

Custom Resources for Fiat

Fiat is the authorization server for Spinnaker

written as a Spring plugin currently, may soon use a Resource extension point that enables serving permissions for a new resource type, such as read, write, and execute permissions for a “File” resource demonstrated in this example

Cloud providers

Cloud providers, in the context of Clouddriver, refer to target platforms where Spinnaker can deploy workloads, such as Kubernetes, EC2, Cloud Foundry, etc. Before the plugin framework, adding a new cloud provider was a notoriously difficult task. We’re expecting to see more of these cloud providers developed as plugins, as these are easier for those with less familiarity with Spinnaker’s codebase.

An example of how a cloud provider plugin might look, at least for the back-end, is illustrated by this one for Hashi Nomad. Some key components:

Caching agents, which are how Spinnaker checks for current state of any particular cloud provider resources

Atomic operators, for updates to these cloud provider resources

Searchable provider, for finding a specific set of resources

A complete cloud provider plugin will also add custom stages (e.g. a custom Deploy stage) and Deck updates to aid with visualizing new types of cloud resources.

Integrations

Plugins also enable more integration points with other tools. This is an improvement over older means of integrating with these tools, such as the Armory Jira stage (which only enables auto-updates to Jira from Spinnaker, not the other way around) or the echo REST webhook, e.g:

datadogEventListener (which extends EventListener ) showcases how plugins offer more control over events sent to a downstream events aggregator (Datadog, in this case), so you can add filtering and transformation logic

) showcases how plugins offer more control over events sent to a downstream events aggregator (Datadog, in this case), so you can add filtering and transformation logic this observability plugin enables customizing the Micrometer registry and exposes an OpenMetrics endpoint for the Micrometer/Spectator metrics, which allows tools such as Prometheus or the New Relic OpenMetrics integration to work without the Spinnaker Monitoring Daemon

integrations with notification systems (those which don’t appear in the notifications drop-down in the stage configuration) can be enabled via a plugin like this one

Custom UI/UX

Lastly, custom visualizations of information in Spinnaker are an ideal application for plugins. Think SDLC audit trails, deployment graphs, interactive dashboards, etc. The community has not yet publicly shared examples of these, but this is a popular customization theme.

Plugin training from April’s Gardening Days. Register for the July Gardening Days at go.armory.io/gardening

Where do these plugins end up?

The community puts example plugins on GitHub. Soon there will be public project(s) hosting plugins meant for production use, such as the observability plugin. Teams can host their own plugin repositories and make them public or private.

How does this impact the Spinnaker project?

The main motivation for this plugin framework, as mentioned earlier, was to move the project towards a lean core model. Alongside this lean core there’ll be a rich ecosystem of extensions enabling a diverse range of features. This makes it easier for contributors to add new Spinnaker services. This does NOT mean Spinnaker will feel less full-featured “out-of-the-box”; plugins add only a few more steps to your deployment, which we cover below.

How do I get started? I want to try using a plugin.

Take a look at our docs. And this video tutorial may help, too.

Join Gardening Days July 16–23 for expert help (plus prizes and recognition) with your Spinnaker plugin and extension projects.

And how do I create a plugin?

For setting up a developer environment for Spinnaker, checkout the instructions here.

For development of a plugin once you have a dev environment ready, take a look at our docs. And these video tutorials from a past Gardening Days may help, too.

If you have any questions or feedback, join us for Gardening or get in touch with us on the #plugins channel - we’re more than happy to talk with you!

Ryan Pei is a Senior Product Manager at Armory.