Improving Developer Experience with Ephemeral Environments
By: Eric Irwin
Tags: Cicd, Gitlab, Continuous Delivery, Engineering, Spinnaker

At Quantum Metric, Ephemeral environments have become a core feature of how we deliver as the size of our teams and complexity of our platform have increased.

Ephemeral environments, also known as “on-demand” or “dynamic” environments are lightweight, short-lived instances of our production environment that can be created for validation, testing and experimentation.

Since Quantum Metric refers to our customers as subscriptions, we instead refer to an environment as a subscription, so for the remainder of this post, we will refer to Ephemeral subscriptions synonymously with Ephemeral environments.

The purpose of this post is to share how we leverage ephemeral subscriptions to improve Developer Experience, and how we plan on continuing to improve the self-serve capabilities we offer to our teams to work seamlessly and effectively when delivering.

Why Ephemeral?

In order to understand our need for ephemeral environments, lets look at the problems and symptoms we were experiencing before we implemented them.

High “Cost of Coordination”

Who is using environment “X”? Can I use it?

If you are frequently asking these questions, then chances are you are suffering from a high cost of coordination — that is, there is time (and therefore cost) associated with coordinating use of an environment.

Based on the amount of environments available, and the size and velocity of the team(s) relying on their availability, the impact of this can certainly differ.

As our team size continued to grow, so did the frequency in which we found ourselves losing valuable time to coordinating use of these environments.

Environment Drift (aka “Rot”)

Who broke environment “X”? Was it me?

Static environments are continuously deployed to and configured in ways that are often out of the control of the person using it at any given time.

This lack of isolation and consistency leads to confusion during validation and testing.

If a bug is found or something breaks, it isn’t immediately clear whether it is due to any particular change, or whether the environment has been left in a broken or misconfigured state.

We recognized this as an increasingly common occurrence and symptom that we were outgrowing the existing solution.

Data Availability

How can I test/experiment/reproduce this?

Our teams have come to rely heavily on the ability to test changes against a wide variety of data because of the diversity of our customers. Not only is this true when testing new features, but also for experimentation and triaging issues.

As our customer base grew, so did the need for us to provide different types of data for testing without requiring to test directly against production (though we are doing this more frequently. See Continuous Delivery and Automated Canary Analysis using Spinnaker and Prometheus)

For a long time, we relied on manually configuring and redirecting data to environments in order to satisfy this need. This led to confusion and ultimately wasn’t sustainable.

How did we solve these problems?

As these problems became more prominent, we knew we needed to address them.

Enter Ephemeral subscriptions.

Starting in late 2020, we committed to solving these problems by implementing a solution that would centralize the creation, management and orchestration of ephemeral subscriptions so that they could be consumed directly in our everyday developer experience.

We were fairly confident this would evolve into something much bigger, so our first iteration we decided to keep it simple and implement this as an opt-in solution for GitLab Merge Requests (MR) since that was frequent touchpoint for majority of the target users.

An opt-in solution would allow an ephemeral GitLab Label to be added to a Merge Request, which would result in the resulting pipeline to trigger the creation/deployment of an ephemeral subscription.

GitLab MR Label

More on this later…

Several iterations later, we have not only delivered on this commitment, but have continued to add valuable features and capabilities to improve this experience for the end-users.

Some of the features we have added:

Using a data MR label such as data:bigco , an MR author can choose to securely seed an environment with data from a specified source (e.g. bigco ) that they would like to test or experiment with.

MR label such as , an MR author can choose to securely seed an environment with data from a specified source (e.g. ) that they would like to test or experiment with. Similarly, live non-production data can be configured from one of three traffic generation sources.

Environment cleanup is automated based on a rolling window of inactivity.

How it works?

Automating the creation of environments to closely resemble production was far from trivial.

Ensuring true isolation and that they are light weight enough that they can be created/destroyed quickly was even more challenging and we are still finding new ways to improve on this.

Below is a high level overview of the components used for creating and managing ephemeral subscriptions.

1. GitLab MR Pipeline Triggered

When an MR is tagged with the ephemeral label, all subsequent pipelines for that MR will be scoped to an ephemeral subscription.

The GitLab CI job is responsible for propagating the following information in an API request to our subscription-manager service which acts as an API gateway for managing inbound requests when creating and managing subscriptions.

Branch Name — The branch name is used along with the Project ID to determine the name and endpoint for the subscription that is created.

For example, an MR for a branch named feature/my-test in a project with an ID of 1 will result in a new feature-my-test-1 subscription being created. This in turn will be added as an A record in DNS so that the new subscription can be accessed.

Since it can take a few minutes for this to be completed, we ensure that we post a message back to the MR to keep the author informed. This is shown in the GitLab Release Bot Notifies MR section below.

Project ID — This is used to ensure uniqueness across projects to avoid conflict in naming. Keying off of Project ID + Branch guarantees uniqueness

MR Author — This is used so that we can continue to notify the MR author when updates/changes are made to the subscription.

Data Label(s) — As previously mentioned, the MR author can use data:* MR labels to signal what data to use when seeding a subscription.

While the high level diagram reflects a CLI component, we haven’t implemented this yet and are expecting it to be one of the next features that will allow ephemeral subscriptions to be created/managed directly through a CLI.

2. Subscription Record Added to Database

When the subscription-manager endpoint for creating a subscription is invoked, it will run through basic validation and ensure uniqueness before adding a record to the subscription table with a pending_creation status.

For convenience, we store additional metadata with this record:

Data Source — We use this to track the source of the data that the subscription was seeded with.

Traffic Type — Similarly used to track the type of traffic generation for sending live data to the subscription.

Last Updated — Used to track inactivity since we automate cleanup based on a rolling window. Every time a change is deployed to this subscription, we update this value to ensure it will not be cleaned up.

Host — Used to track which host the subscription was provisioned on so that we can load balance creation of new subscriptions based using a Least Used strategy.

Status — Used to track the status of a subscription during creation and cleanup. Since the process is asynchronous across many moving parts, this is used to ensure that we always have visibility into subscriptions that are not in a ready state.

3. Create Subscription Message Dispatched on PubSub

Once we have added this record to the database, we dispatch a message on a subscription-requested PubSub topic reflecting that a subscription was added.

The payload for this is pretty straight forward and includes the necessary information for downstream components to handle creation and configuration for the subscription.

{

"name": "feature-my-test-1",

"author": "jsmith@mysite.com",

"data": "bigco",

"traffic_type": "demo",

...

}

The decision to use PubSub was based on the recognition there would be future components that needed to be notified, or listen for events.

Using PubSub limited the possibility we would have to explicitly invoke additional endpoints since we could simply listen on the topic(s) we were publishing to.

4. Spinnaker Create Ephemeral Subscription Pipeline Provisions New Subscription

As we mentioned in Continuous Delivery and Automated Canary Analysis using Spinnaker and Prometheus we have adopted Spinnaker as our Continuous Delivery Platform.

Spinnaker’s robust pipeline capabilities fit our needs for orchestrating the creation and management of subscriptions and since our Spinnaker instance was already configured to use many of the common components we needed for this (e.g. Cloud Storage, Cloud Database, etc) it made sense to leverage it.

Within Spinnaker, we have a Create Ephemeral Subscription pipeline that is configured to trigger when it consumes a message from the previously mentioned subscription-requested PubSub topic.

Create Ephemeral Subscription Spinnaker Pipeline

Out of all the steps required for creating new subscriptions, execution of this pipeline is the most time intensive and requires the most care to ensure that race conditions and intermittent errors due not leave a subscription in an unhealthy or partial state.

Under the hood, the Create Ephemeral Subscription pipeline has quite a bit of moving parts, but its primary responsibilities are:

Provisioning new subscriptions to the provided host and ensuring all of the services are properly configured and healthy.

and ensuring all of the services are properly configured and healthy. Seeding subscription data based on the data source provided in the payload.

source provided in the payload. Configuring traffic generation based on traffic_type provided in payload.

provided in payload. Adding DNS records for subscription endpoints.

Notifying progress and updates for subscription creation back to the owner.

5. GitLab Release Bot Notifies MR

Once the subscription has been successfully created, Spinnaker will update the originating MR on behalf of a GitLab Release Bot that the subscription has been created and is ready to be accessed.

GitLab Release Bot Updates Merge Request on Ephemeral Sub Creation

The purpose of this step is to ensure we are tightening the feedback loop for the author(s) of an MR and they are in touch with the availability of their environments.

We have found this pattern to be useful, and use it to automate feedback on MRs such as when an MR has been open for too long, or whether it might be missing details that will result in it not being able to be merged and released.

Future Improvements

As we continue to observe broader adoption both within and outside our Engineering teams, we will continue to drive towards an improved development experience for our end-users.

Some future improvements we are planning on making:

Pooled Subscriptions — In order to reduce the time for creating a new subscription, we would like to configure a pool in which pre-baked subscriptions can be readily acquired. When that subscription is no longer needed, it is still destroyed to prevent environment rot, however a new fresh subscription can be created in its place.

— In order to reduce the time for creating a new subscription, we would like to configure a pool in which pre-baked subscriptions can be readily acquired. When that subscription is no longer needed, it is still destroyed to prevent environment rot, however a new fresh subscription can be created in its place. Data Availability — While we have made many steps in the right direction, there is a lot of room for improvement to allow end users to pick and choose specific data scenarios they would like to test and experiment against.

— While we have made many steps in the right direction, there is a lot of room for improvement to allow end users to pick and choose specific data scenarios they would like to test and experiment against. Performance — Reducing the amount of time it takes to create a new subscription means we are allowing quicker time to validation and don’t run the risks of race conditions related to our CI/CD pipelines.

— Reducing the amount of time it takes to create a new subscription means we are allowing quicker time to validation and don’t run the risks of race conditions related to our CI/CD pipelines. Reliability — With increased adoption, there is also increased dependency. With so many moving parts, we need to continue to improve resiliency to ensure that end-users can continue to rely on these subscriptions.

— With increased adoption, there is also increased dependency. With so many moving parts, we need to continue to improve resiliency to ensure that end-users can continue to rely on these subscriptions. Configuration Management — Similar to data, we would like to improve the ability for end-users to manage configuration directly.

Conclusion

Ephemeral environments have changed the way many of our teams work and interact. They have opened up opportunities and exposed capabilities that we didn’t previously consider possible.

Similarly we have stumbled across new challenges that we didn’t previously have to deal with. When you alleviate one bottleneck, there often is another one waiting for you.

Some interesting facts on our current usage:

More than 1000 ephemeral subscriptions have been created YTD across 30–40 participating engineers.

Ephemeral subscriptions have been used across other non-engineering teams for experimentation as well as incident management and reproduction.

Rate of adoption continues to increase.

We have observed a 1–2 day decrease in Mean MR Lifetime (time taken for an MR being open to being merged and released).

Thank you to all the teams and contributors that made this possible.

If you found this interesting and want to hear more about what we have in store for the future, our Delivery Engineering Team is growing quickly. Definitely check out the opportunities we have available.

There are also many other exiting roles we have open, so don’t hesitate to visit Quantum Metric Careers