The Azure Journey: Configuration Management
By: Tim Ellison
Tags: Azure, Azure App Configuration

Somewhere in Aruba — 2018

If you’ve been in the IT business on the technical side for any length of time, you’ve most likely encountered at least a few different ways to manage configuration for applications. Taking a look backwards, I’ll share some of mine but then we’re going to take a step forward and explore a service based approach to it called Azure App Configuration.

Before we get started, we should at least seek to understand the goals behind configuration management. The fundamental core of this practice is simply to manage the configuration differences for applications as they are deployed from one environment to another. While it may appear on the surface that CM and Security are the same, their relationship tends to be intertwined. CM cares about practices that help manage the settings (e.g., connection strings) of applications whereas Security cares about protecting those settings from people who shouldn’t see their values. To keep this conversation a little higher level, we’ll confine “Configuration Management” to the management of a connection string for a database server. We’ll add one additional item to the mix; Feature Flags.

Feature Flags

No organizational DevOps practice is complete without the incorporation of feature flags. Feature flags at their heart are simple bit values (On or Off). Their goal is to promote “roll-forward” deployments as opposed to “rollback” deployments. In practice, rolling an application backwards has as much or more risk than rolling it out. An application with any complexity will typically cost both time and budget to roll back. Time-consuming rollback impacts the bottom line either in the form of application management cost or lost revenue.

On one engagement, we were asked to roll out a new feature to improve performance. The underlying service had been improved to use Azure Search to return data.

Widgets Sell!

In “the old days”, we would have refactored the service and rolled it out hoping for the best. With a feature flag in place, we could roll out a new version of the service keeping the original in place until we turned on the feature flag. The size we picked for Azure Search couldn’t handle the number of requests. When we turned the flag on, the queued requests metric shot through the roof threatening downtime (lost sales). We quickly turned the flag off reverting the application back to the slower but more stable former version of the service. This allowed us time to resize Azure Search and then turn the feature back on for vastly improved performance.

Historic Configuration Management

Configuration Management has evolved over the years (decades now) from completely manual to some degree of automation. I thought I’d share some of the “flavors” I’ve seen over the years. Some of them were the best we could have at the time while others are borderline “questionable”.

Human Touch Deployment

One of my least favorite approaches to managing configuration is the manual approach. As much as we like to believe we are machines, we humans aren’t. We are exceptionally error prone. That said, we are not necessarily the best option for repetitive tasks like configuring connection strings. My favorite adventure was discovering a fully manual approach where developers would compile the code locally then attach the compiled binaries to an email. The email was received by a “Configuration Manager” who followed the instructions in the email to deploy changes.

This really happened!

Configuration Deployment Avoidance

This is another approach I’ve seen where configuration is all performed on the application servers (or app services). It’s another of the manual deployment approaches that introduces the risk of downtime because a human (remember we’re error prone!) is responsible for ensuring that the new connection string is available on the application server. While it makes the inside attack surface smaller, it almost always results in failed deployments. As well, it makes upgrades challenging. It’s not unusual these days to add assembly binding redirects (ABR). A number of packages available from nuget all have dependencies. When we upgrade one of them, an ABR is needed to ensure that requests for version 1.2 can be redirected to version 1.4.

In my personal opinion, this type of CM is a born of laziness more than anything else.

The All Hands Deployment

Another flavor of Human Touch Deployment is the “All Hands” version. We used this approach at a company I worked for. Every quarter, the entire company committed to a weekend of rolling out the application. Think Human Touch Deployment on a massive scale.

I remember waking up Saturday morning under my desk, wondering when I was going to die so that it could just end.

The Analog Safe

Most of us these days are using Azure Key Vault to varying degrees. One of my clients used the analog version of Key Vault and kept all the application secrets in an envelope in a safe he had bolted to the floor of his office. This particular application was a first foray into service oriented architecture. The application front end was a very thin client with a number of services supporting each of its respective features. A deployment required an additional trip to his office so that he could open the safe and get out the secrets which he then applied to each service following the deployment.

Compliments of HarborFreight.

Multiple Web Configs

Before Microsoft introduced Web Config Transformations (and a really cool tool called Slow Cheetah extended it to all XML configuration files), many of us stored web.[Environment].config in source control. Either a custom build server task or a manual post-deployment task would then take the appropriate configuration file and rename it to web.config.

Configuration Transformations

I don’t recall when Microsoft introduced these but a quick “Google” reveals that configuration transformations were introduced in 2012. At build time, the msbuild engine would take the configuration (Debug, Release, etc) and apply XML transformations from the respective web.[configuration].config to the web.config file. This approach moved the needle further away from human touch deployment to the end goal “No Touch” Deployment.

Protecting Secrets

With the exception of the more centralized human approaches above, all of the aforementioned approaches exposed the configuration of upper environment application secrets to source control and invariably to all the engineers, testers, project managers and administrators of source control. For some applications, this isn’t a huge security concern. For example, a CMS that shows content to anonymous end users isn’t a risk. At least, it isn’t until that employee who got fired decides to get revenge and replace all the media assets (images) with images of body parts in various states of undress. This actually happened a few years back and No, I wasn’t the person who did it but my clients’ brand image was severely tarnished resulting in lost revenue. Ultimately, ANY configuration value that results in access to back ends can create the potential for a security breach. Many membership providers capture the bare minimum for registered users usually in the form of email address, name, phone number and password. This is quite often enough information for a “bad guy” to start down the path of impersonating the user. Many of us have bad habits on the internet and will often use the same password across sites. This combined with the capture of membership data opens the door for identity theft or at bare minimum, some embarrassing pictures on facebook.

Environment Variables

Applications that rely on environment variables for their secrets is a great first step towards protecting an application. They offer ease of setup from the development environment to production. Unfortunately, they still offer people with enough credentials to see them and they are typically stored “in the clear”.

DevOps Tokenization

Moving configuration to a build server is another widely used approach. Instead of storing the values in environment variables, they are stored in a list of key/value pairs and typically replace tokens stored in configuration at build or release time. Without encryption though, the values are still stored on disk in the clear and anyone with access to the server can see the values once they’re deployed. Even moving secrets to Azure Key Vault and exposing them at build/compile time results in the same thing, “in the clear” data at rest.

Encrypted Configuration

If you’re going to use configuration transformation, this approach helps ensure that prying eyes can’t see the upper environment configuration in the clear within source control.

Azure Key Vault

Key Vault moves the management of secrets (our connection string) to a secured API. With access policies, we can then secure the secrets from prying eyes and the data is encrypted at rest. If you pick any of the non-encrypted approaches above though, you’re still exposing your connection string.

Azure App Configuration

Azure App Configuration is a “newish” service offering from Azure. Per its description “Azure App Configuration allows developers to store, retrieve and manage access to application settings all in one place. It is easy to set up and simple to use from any application. It gives developers the ability to modify an application’s behavior on demand without having to redeploy the application. App Configuration offers the following benefits:

A fully managed service that can be set up in minutes.

Flexible key representations and mappings.

Versioning with labels.

Point-in-time replay of settings.

Comparison of two sets of configurations on custom-defined dimensions.

Enhanced security through Azure-managed identities.

Complete data encryption, at rest or in transit.

Native integration with popular frameworks, including .NET and Java.”

In short, app configuration is your “one stop shop” for configuration values. One key take-away here though is the bullet “enhanced security through Azure-managed identities”. Managed identities (V User Identities) are tied to the app service. If the app service is removed the managed identity goes bye bye as well. If your shop is a stickler for infrastructure as code (provisioning via ARM templates) this may not be a problem for you.

When you’re planning your system architecture, one thing you’ll need to think about is how much you’ll want to mix settings within the same App Configuration service. For higher availability on larger architectures, you may want to consider having multiple app config services. In fact, Microsoft’s documentation on Best Practices suggests a configuration instance per region.

Simple Key/Value Pairs

App Configuration supports key/value pairs (similarly to Key Vault) although per the documentation, it doesn’t recommend storing secrets. You need to decide what a secret is. An API key that allows full access to the underlying data is most likely a secret whereas an email address we send from “donotreploy@contoso.com” is most likely not. Use your best judgement.

Azure Key Vault Secrets

Secrets are best stored in Azure Key Vault. Azure App Configuration supports storage of keys that are a reference to a key vault secret. Behind the scenes, app config just stores a json string that contains the URI to the secret from key vault.

Feature Flags

Feature flags are awesome. Period! When I was first introduced to them, it opened up a whole new world for me. I’ve already talked about it above but it’s helpful to reinforce it here. If you’re not using Feature Flags in your application, you may want to take some time and reflect on your Engineering and DevOps practices.

One thing to remember with feature flags when you’re retrieving them, regardless of the Name you gave them they are case sensitive and the name you provide is preceded by “.appconfig.featureflag/” as a namespace. The result is a json string consisting of id, description, enabled and an array of filters. The filters support advanced use cases such as enabling a new feature for a select group of users. One potential use although I haven’t experimented with it could be something similar to A/B testing or a rudimentary form of personalization.

Static V Transient App Configuration

Based on the needs of your application, you’ll need to determine which settings will hold constant (static configuration) for the lifetime of the application and which should immediately be reflected when app configuration changes (transient). This will determine for you whether you’ll use Config Builders or access the App Configuration via its API. Connection strings (secrets) aren’t likely to change for most applications so they can be relegated to a config builder and accessed when the application starts. The aforementioned “from” email address is most likely another one of those types of settings that’s more static in nature.

Sample API Code

Simple Feature Flag model (minus filters)

Simple Synchronous API client

For settings like Feature Flags, you’re almost always going to need to access them via the API. The sample above, while not production quality, serves as an example of how simple it is to interact with the service.

One thing to be hyper aware of is the cost surrounding excessive querying of the service. The section entitled “Reduce requests made to App Configuration” in the Best Practices link above talks about this in detail and offers some examples of how to avoid “over-querying”.

Take-Aways

If you’re considering improving your DevOps/Engineering practices, I urge you to take a serious look at the App Configuration service and at Key Vault. Additionally, think about how you’re going to release that next feature and what can happen if it breaks the application. There are great articles out there that talk about Feature Flags and how to incorporate them into your application. That knowledge combined with storing them in App Configuration can be transformational for your practices.