Inversion of Control for the Infrastructure with Consul
By: Pierre Souchay
Tags: Software Development, Consul, Infrastructure, DevOps, Prometheus

In the first article Discovery at Scale With Consul, I explained how we added support for Service Metadata to achieve automatic provisioning of Load-Balancers (LB) in our infrastructure, allowing us to create a full stack of Reverse proxies in a declarative, explicit and decentralized way. Let’s go a bit beyond this usage and see what we can do with metadata.

Enriching Consul with declarative data

Consul lets you enrich your agents and services with additional data. Let’s start digging into what is possible and how we enrich that at Criteo.

For each node

A Consul Agent is running on a node (it can be a VM or a real machine) and you can enrich it with metadata. This is just Key/Values (K/V) of strings that give more context about the node your agent is running on. At Criteo, we thus enrich the node description with information such as:

the owner of the node (which team is handling hardware issues for instance)

the physical location of the node

the operating system installed

the flavor of hardware (i.e. what kind of node, CPU…)

All declared metadata are then available when requesting any of the services declared on the node.

For the services as well

But of course, this is not enough, we need some information about services as well. In Consul, a service is just a name that allows retrieving instances with the same semantics, aka performing the same job on the same set of data. For instance: `sql-mariadb-billing` might be referring to a SQL service speaking mariadb SQL dialect and handling billing information, while `web-static-js` might be composed of ngnix and apache webservers serving Javascript assets to your customers.

Naming is always hard and you have to clearly define a strategy to name your services. With more than 4000 kinds of services, we had to find a way to deal with that and use Consul ACL tokens with prefixes. Each team in the company has a set of tokens assigned to it and the prefix is the namespace. A Token As A Service provides to the teams a way to request a new token that does not conflict with other teams prefix, so we are sure every team can decide the names of its services in autonomy (eg: `sql-mssql-` prefix for the database administrators of MS SQL servers).

In any case, while the service is the same, `sql-mariadb-billing` service might be handled by a MariaDB or a Mysql server, while web content might be served by different software products. Even if the main service is the same, we might end up dealing with different tools to administer those services or retrieve metrics (think the metrics exposed as Prometheus endpoints might be different or not on the same path for nginx and Apache httpd).

Initially, Consul only supported tags for services, a list of strings added to a service that allows filtering for a well-known keyword. The only downside is that you have to know what you are looking for, meaning that if you decide to use a tag `server-type-nginx` or `server-type-apache`, we will end up with horrible regular expressions everywhere to deal with the cardinality of these tags. Furthermore, nothing prevents that a service might end-up with both `server-type-nginx` or `server-type-apache`, so how to deal with these ambiguities?

Service Metadata, on the other hand, allows to specify a single value per key and get precise semantics. ”version”: “42” is single-valued and while it is possible to create tags `version-42` and `version-43` at the same time, service metadata allows for ambiguity free declaration. Moreover, tags are limited to some characters (notably because they can be used in DNS requests), while service metadata values are not.

The rule is the following:

if you need to search for features (e.g.: ` http `, ` monitoring `), use tags with well-known tags

`, ` `), use tags with well-known tags if you need to get precise information, use service metadata instead (eg: version, path…)

Initially, before the implementation of service metadata, we saw some implementations using the Key/Value store to save this kind of single-valued information. We recommend avoiding the K/V as you would have to perform cleanup by hand or might have issues dealing with instance cardinality (For example, pushing version information is probably a bad idea since several versions of the same service might co-exist at a given time).

What is IoC? (Inversion Of Control)

The idea behind IoC is to decouple the various tools and systems in the infrastructure. When you want to monitor a service, the usual way of doing this is to dig in the configuration of your favorite metrics collector system. In the long term, it does not scale very well: you’ll end up with massive git-based repositories configuring all your checks, load-balancers or alerting systems without any strong link between the service it is operating on and its configuration.

Using service/node metadata allows pushing configuration of foreign additional services embedded within the service or node it intends to configure. In our case, we offer to the teams managing a service the ability to tune the tags and metadata, we thus have a complete decentralization of tags/service meta, so stakeholders can act quickly by changing these metadata without relying on other teams, at their own pace.

Service and node metadata are automatically cleaned up when services/nodes disappear, so the cleanup is automatically performed when such data is not relevant anymore.

All of it happens in real-time as Consul offers a watch mechanism we use extensively to be notified when anything changes in the Consul’s catalog. We improved and optimized quite a lot this mechanism over time (examples of our Pull-Requests on this subject: #3899, #4720, #4810, #5858… see the impact on only one of these patches), so it is now very efficient and Consul API notifies you only when needed (with the optimizations we merged, watching all services on Criteo’s large clusters went from 800Mb/s to 16kb/s ). So, with recent versions of Consul, you can use it for free without overloading your Consul cluster or bandwidth.

Inversion Of Control with semantics

In complex infrastructures, the coupling between various systems is difficult to avoid and many companies are paying a high price for this coupling (sometimes called Technical Debt) when systems have to be migrated to a newer and fancier system.

For instance, a few years ago, Criteo moved away from Centreon to Prometheus / Alertmanager. At that time, the migration took ages (something like 2 years), what if we could avoid paying this price twice?

As people do not care about the third-party system monitoring of the moment, people don’t want to learn about configuring each third-party service interacting with their system, they just care about their business and constraints. Business constraints might have semantics such as: “As a service owner, I want to be notified on my slack channel when I lost more than 50% of my instances for some reason for more than 15 minutes”, this could be translated by something like:

How metadata for alerts can be expressed

Expressing business semantics is the key because it will express clearly the features of your system instead of exposing all possible configuration options and creating an unnecessary coupling between systems. It also creates a clear contract between different teams, so these interfaces can be implemented by different service provider implementations.

How does it work?

As explained in our first article, some additional services providers will listen to all changes in the Consul services.

Service Providers listen to all changes in Consul and create additional services based on those changes

Each time a service is created/deleted, a provisioning operation is applied to reflect what is happening in the data-center. Such operations can be:

adding a new service

adding/removing some instances of a service

once all instances are removed, remove the service

Opt-in Versus Opt-out parameters

The systems providing new services based on metadata can perform choices. For example, our automatic alerting system is Opt-out, so you have to specify you are not interested in it if you don’t want to use it.

It also means that providers can perform choices based on metadata and choose the values they want when these are not specified. In our example, by default, our alerting systems start complaining when you lost 2/3 of your instances unless specified as metadata for the service.

Being able to use default values also decouples service from the infrastructure as the service providers can choose default values that make sense for most of the customers.

Some Real-life Examples

Provisioning of Criteo’s Load-Balancers

See our first article.

Swagger Catalog Application

This app watches all services exposing a swagger catalog (exposed with a “swagger” tag) and analyzes the metadata. The app can then display the endpoints with documentation, authentication, and the ability to call the endpoints.

Service Metadata is used to know the path to swagger as well as configuring authentication, find documentation

Metrics of Availability with alerting for 100% of Criteo’s Services

All services availability is computed from Consul Health information using a very simple Prometheus exporter template from consul-templaterb that exports metadata as well, so you can at any time know how many instances are healthy/warning/critical with their version on all the data-centers. Metrics are scraped by Prometheus and sent to Grafana dashboards.

Service Metadata is then consumed by templates to build automatic alerts based on metadata and wakes-up on-call team when needed or sending messages on Slack Channels for alerts.

Service Availability Metrics for free!

Ownership issues detector

This application looks at all services and allows finding all applications not having a correct owner (for instance, the team does not exist anymore or has less than 2 people). It can also detect applications not deployed for too long.

ownership / old versions detector

And many more…

Creating such an application is easy, write a template, read the file and act based on basic rules.

For instance, during HashiConf 2019, the demo made at the end of the presentation was using this template: https://github.com/criteo/consul-templaterb/blob/master/samples/demos/compute_pricing.txt.erb.

On our infrastructure, this would create the following content:

— -

summary: Price for web-.*

machines: 8053

price: 94735

flavors:

base-10g-2015: 3290

base-10g-2018-q-1: 2890

base-10g-2018-q-2: 468

base-1g-2013–1: 522

base-1g-2015: 472

ram-10g-2018-q-1: 7

ram-10g-2015: 85

base-1g-2013: 288

ram-1g-2015: 2

storage-10g-2015–1: 29

unkown_prices_for:

base-10g-2018-q-2: 468

Such a simple template (~10 minutes to write it) computes in real-time a virtual cost for all the agents running a given set of services.

Such kinds of templates use Consul as an always up-to-date database and enable our users to dig into the infrastructure, experimenting fast and extracting important information for applications and infrastructure.

How IoC change the way we operate a platform

Semantics enforce innovation

Since we defined business semantics in service’s metadata, anyone can quickly develop competing tools to replace infrastructure from their workstation: it is now very easy to create a new kind of alert manager without disrupting operations on existing ones. You just have to implement the same business semantics. You don’t have to modify anything in the 4000 different kinds of services at Criteo.

Enriching services with metadata is a virtuous cycle

Each time someone adds some metadata (e.g: version, artifact ID…), this allows creating new services (e.g: direct access to change-log for this version).

These data can then be re-used to create new services using the same semantics.

Pushing metadata ease onboarding of new-comers

All configuration is located on the service, so it is very easy to see all configuration of service, jump to get details and get metrics, logs…

Here is the annotated UI displaying information for a given service instance:

service instance details with meta and additional services built on top of it

The UI reflects all the metadata available for all services and allows accessing all necessary information very easily. It also includes all configuration for load-balancers, creates links to access the logs, the metrics, automatic alerts and gives a very useful context for new-comers. Integrating new people at Criteo might be thus more efficient more quickly.

Something As A Service VS Inversion of Control

The fashion is to do Everything As A Service, in the same way as cloud providers do. This enables very quick iterations and removes the need for ticketing based provisioning (which is good). This is quite nice and enables different teams to build services on top of services.

But Inversion of Control is more powerful than that: it builds magic things. As soon as your service is available, everything is properly configured. Of course, it does not mean that your API of Services is deprecated, it simply means another system will use it. Once again it reduces the coupling between your applications and the business services you are running.

People can focus on their business issues

Since you just push business semantics (such as “I want my application to be published by DNS name myservice.criteo.com”), people don’t have to know everything about the infrastructure (whether you are using bind or PowerDNS is not very interesting for business). It means engineers can focus on solving real business issues rather than dealing with changes in the infrastructure.

Furthermore, this decoupling allows upgrading more easily the infrastructure.

There are no more cleanup tasks

Since all data lives within the service, all configuration is cleaned up automatically once the last instance dies. There is no other cleanup to do, the life-cycle of monitoring systems, Load-Balancers and other systems is linked to the real life-cycle of the application.

Ownership is public and well defined

When a company is big enough, ownership can be hard to maintain. We are using service metadata to enforce it. Now every service has an owner that can be contacted when issues occur. In our case, ownership is linked to LDAP groups and is enforced continuously. We thus verify that each service has an owner defined and this owner is both valid and includes at least 2 people. Whenever the ownership rule is broken, someone gets alerted to fix the situation.

This enables us to have a much more effective response in case of issues with any service Criteo is operating.