The cost of 10 lines of code
By: Anton Malinskiy
Tags: Engineering, Nexus, Docker, High Availability, Kubernetes

Sooner or later, every team needs artifact storage, e.g.: Maven, npm, docker, etc.; preferably with caching support. Having in-house storage can dramatically improve performance. For example: cold builds of Android applications or APK installs in Alpine can be sped up by tens of minutes, depending on how slow your internet connection is.

The problem comes when you want to have SLAs set for artifact storage system: e.g., you need 99.9% uptime. This is where High Availability (or HA) comes in. Even though you don’t have a lot of control over the quality of a particular system, you can implement safeguards: health checks and additional replicas in case some of the replicas misbehave.

Unfortunately, most of the solutions currently on the market require you to pay a lot of money for an implementation of HA ([1] or [2]), but there are open source versions of these systems which we can build our solution from.

Architecture

Let’s see how we can design an HA deployment of artifact storage. The two possible replication strategies that exist are replicate-on-write and replicate-on-read. The first (replicate-on-write) is complex to implement, but provides strong guarantees about consistency. The second (replicate-on-read) assumes that the system automatically adjusts according to client requests: if something is used a lot by users, it will end up in the cache. Of course there is a trade-off here: the time it takes to get something in case of a cache miss is increased.

We want to start small, so next we will try to design an artifact storage system and implement the replicate-on-read strategy.

Diagram of a lonely repository

This is where we start our journey: we have one instance of a repository and everyone accesses the only available instance. Suppose it’s an internal Maven repo, or a docker repo. The specifics of which protocol we’re using doesn’t change the fact that if the repository goes down, everything breaks.

So let’s split this repo into two:

Diagram of a repo with a write-buddy

Now we have a read-only repo and a read-write repo. The upload of artifact should happen in the write repository, and when the user requests this artifact it is fetched from write repository into the read-only repository. After this, the read-only repository should have a cached copy of the artifact and it doesn’t really need the write repo at all.

A curious reader will probably notice that all of this is true only if the artifact doesn’t really change. This is true for all of properly uploaded releases except, for example, Maven SNAPSHOT’s which are always fetched from the remote repository.

Unfortunately, this deployment strategy still doesn’t have HA properties. For example, if our read-only repository goes down you can’t fetch anything, even though the write repo is still available. So let’s add more replicas.

Multi-replica setup

Finally, we made progress: now, even if one read replica goes down, the system still works, assuming the write replica is still there in case of a cache miss. But there is still one issue: the read-only replicas are not aware of each other; that is, if the client requests the same artifact from all read-only repositories it will be downloaded from the write repo again, and again, and again… So how do we get around this problem? We basically want to create a quorum of read replicas to check and fetch the desired artifact from ANY of read-only repositories. Otherwise we have to fetch from write repo.

All-to-all access replica setup

Changing the code

What we just described basically looks like a message queue. Each repository should respond whether it has the artifact in cache. The problem is that we’re limited by the functionality embedded into the repository software. We have:

Hosted repo (this is our write repository, basically)

Proxy repo (this one will fetch from local cache if present, otherwise will fetch from remote)

Group repo (this type basically groups other repo types into an ordered sequence)

Long chaining of requests during the resolution of dependency

We don’t want the actual request to go to the real source for performance reasons: it’s much faster to check all the other replicas’ caches first. The proxy repo sounds like something we want to use, but it has one flaw: when you set it up you have to provide the write repo, and in case something is missing in the local read-only repository’s cache it will always fall back to the write repo. So, if we specify each Nexus to proxy each other, then the following is possible: suppose we have a request that is coming from a Load Balancer to the first replica, and it just so happens that there is no data cached. It then goes to the source: the second replica. The second replica has the same problem and accesses the third, and so on. This can lead to a long chain of requests — in the best case scenario—and a loop in the worst case:

Looping requests during resolving the dependency

So, basically we want to change the logic as follows:

If the current Nexus is accessing another Nexus’s proxy repository we want the second one to be aware that it’s being used as a read replica If the current Nexus is requested to check the contents of a proxy repository, then we want to check only the cache, not the write repository

These requirements lead us to a couple of ideas:

We can use the User-Agent header to identify whether we’re being accessed by another Nexus instance We need to modify the code so that if we’re accessed by something with User-Agent of Nexus then in the proxy repo implementation we should only check the cache and return early. The actual code change will look something like this:

Code snippet for checking the User-Agent header

After checking the cache we just verify the User-Agent header, which for Nexus should start with “Nexus/…”. Unfortunately, if we just return null, the negative caching will take over next time: suppose we first get a 404 response by only checking the cache. The negative cache will store this, and even though the next time we might want to check the write repository, it will still return 404 for the TTL duration of of the negative cache. This is not what we want: each time the other Nexus’s cache is checked it should bypass the negative cache all together.

After building and checking that all tests are passing, we have a binary and a docker image.

Auto-configuration of Nexus OSS replicas for proxying

Nexus provides us with three kinds of repositories. Let’s recap:

Proxy Repository — it will request and cache from some other repository. This is what we need for our Read-only Repository.

Hosted Repository — it allows us to upload artifacts. It is the main component of the Write-only Repository, and we will not use it for the Read-only repository.

Group Repository — it is a way to join any repositories under one URL. When clients request some artifact from a Group Repository, the Group Repository will go through the list step-by-step, and in order.

So every Read-only Repository in our HA system will have the following Proxy Repositories:

One for every Write-only Repository. This will expose the original versions of our internal artifacts

One for every external repository, such as Maven Central or JCenter

One for every Proxy Repository on other Read-only Repositories of our HA (we’ll name them “brothers”). This will supply us cached copies of artifacts

All of these Proxy Repositories will be joined into one single Group Repository. For external sources it is faster to request “brothers” before the original repo; it will save time and external traffic. For internal (corporate) sources we’d like to request the original first.

The number of Read-only Repositories in HA is your choice, but for the story we’ll choose three. In the picture below we’ll show only two of the three brothers so as not to turn the picture into a mess.

Example of proxying jCenter

Automating the configuration of Read-only Proxies is vital: no-one wants to manually configure three instances of Nexus. Fortunately for us, Nexus has an API for loading scripts. Each script has an API to create and manage different repositories. These scripts are written in the Groovy language. When you execute the script, you can send a data file to it. Examples of what you can do with these scripts can be found in official sources. [1] [2]

So, configuring the system will be split to two parts:

The external script prepares the list of created repositories and passes it to the internal (loaded into the Nexus) script A script inside the Nexus creates repositories according to the prepared list

If we decide to use YAML as data file format it will look something like this:

Configuration example

In this case the loaded script can be as follows:

Configuration script example

Repositories of other types (RubyGems, Docker, blob etc.) are created exactly the same way, but for each of them there is a separate function in the API with a slightly different set of parameters.

Uploading and running internal scripts on Nexus can be done through the Rest API.

Since we really wanted to use a single programming language and get rid of even bash and curl, we decided to use Groovy for external scripts as well and took this example and extend it:

External configuration script

The goal of the external script is to take a configuration file, find information about the location and current state of all Read-only repositories, and produce a list of necessary repositories for the internal script.

To automatically maintain HA systems we use Kubernetes. A detailed description of configuring Kubernetes clusters is beyond the scope of this article, but for us the following is important because of this decision:

Each instance of Nexus in Kubernetes is a Pod which can contain a few Docker containers. In our case, we will have two: container with Nexus and a container with external script.

The exact count and location of all Read-only proxies can vary since pods can migrate and scale; the script cannot influence this, but can receive information about the current state.

Kubernetes can provide a configuration file and other parameters necessary for the execution of our containers. This allows you to separate the configuration from the code and avoid hard-coding URLs, passwords or links.

Kubernetes provides a fairly advanced API. Instead of implementing the calls by yourself, it’s easier to use an existing library. Since Groovy is interoperable with Java libraries, we can use a Java library, for example.

The Kubernetes API provides the ability to monitor the status of all containers. Please note, that the callback will be called in the worker thread.

Kuberenetes API monitoring

The received messages do not contain the new state of the containers, they’re merely a signal about a possible change. Therefore, when you receive an event you need to check whether it reports a significant change for us, and when it comes up, create a new config and send it to the internal script.

Searching for “brothers’ through k8s API

One subtlety worth pointing out: Nexus can take a long time to start, so before you begin to configure it you need to make sure that it has finished booting up.

There is no other know-how to create an external script; you can create your own configuration files and scripts based on your needs. The above code will produce an output which will later be converted to Read-only Repositories. A Group Repository will be created, and in our case it will proxy a resource named “maven-releases” from the Write-only Repository, and a few external resources like MavenCentral and etc.

We packed all the scripts for configuration into a docker image since we found that the Groovy script resolves the dependencies for a long time (more than 10 minutes) and does not always finish successfully.

Script’s Dockerfile

Do not forget to use the same directory with dependencies when running the script (note the argument -Dgrape.root=…)

init.sh

Thanks for reading! We hope your builds are now significantly faster!

This article has been co-written with Igor Tceglevskii