A case for akka-coordination lease
By: Michał Ostruszka
Tags: Akka Persistence, Scala, Kubernetes, Akka

In a project I currently work on we have to run only one instance of a particular service. Basically it’s a service reading a stream of akka-persistence events and building a kind of read model out of those. There is a single events stream and each “entity” follows certain state changes that result in database updates on the projections side. Sounds like a typical Singleton pattern that can be programmed around e.g. Akka’s cluster singleton, right? Yes – and no at the same time.

unsplash.com

The case is that while the service we want to have a single instance of is indeed logically part of an application that is already using Akka Cluster, it is not really justified to put this app in the cluster just to make use of Cluster Singleton. The application isn’t expected to use any other clustering goodies.

Exploring available options

First thing to note before we dive into research: if you decide on using a Singleton pattern you should be aware of consequences and limitations you introduce by that: scalability and availability among the others. Obviously you can’t scale a singleton in a way other than adding more resources to it but this can only go so far. Regarding availability, you need to consider and prepare for cases where the app is simply not available for some timespan. E.g. if it fails due to an OutOfMemory error, there is a time required to detect it and restart/bootstrap a new instance as there is no sibling instance that can take over the load.

Ok, let’s move on to some options available.

As we’re lucky to be on Kubernetes regarding our deployment, the first idea would be to just use its ReplicaSet with a number of replicas set to 1. It’s ok when things are fine, there is just 1 instance of the app running. It starts to complicate when there is e.g. a rolling update in progress or some network instability happens. Take for example a scenario when there is an app running and for some reason it stops responding to readiness/liveness checks. In this case k8s will start another pod assuming the original one died for some reason. When after some time the first pod comes back, there is a slight timespan during which there are two instances of the app running until k8s sorts it out and kills one. This is because a ReplicaSet works in a way where it guarantees the “at least”, not “at most” or “exactly” in terms of the number of replicas running at any given moment.

Another Kubernetes-based option would be to use a StatefulSet which can provide stronger guarantees in terms of pods lifecycle, meaning it will not start new pods until the previous ones gets shut down completely with regards to desired number of pods. At the first sight it looks fine as it doesn’t violate singleton guarantees, but there are some limitations of a StatefulSet that make it quite complex to use (volumes management, headless service requirement etc.)

You can finally try managing the singleton-like behavior on your own, e.g. by instances updating some persistent entry to control which running instance is in charge of being active and which one is in kind of “standby mode”. But that’s a lot of infrastructure-related code (track current active instances, record heartbeats, manage timeouts in case of failures etc.)

This sure looks like a good candidate for a reusable piece of code/library to use and this is where akka-coordination and its Lease concept take the scene. As said on their doc:

The lease is a pluggable API for a distributed lock.

and

Any lease implementation should provide the following guarantees:

— A lease with the same name loaded multiple times, even on different nodes, is the same lease

— Only one owner can acquire the lease at a time

The only available (official) implementation at the moment of writing is kubernetes-lease. Looks like a good match for our use case so let’s review general lease usage in more details.

First you need to create lease instance using LeaseProvider and by passing lease name, config key that contains this specific lease’s configuration (implementation dependent) and the owner of the lease which should be unique for all instances working with given lease (hostname and port of the ActorSystem combination is a good default here).

val lease = LeaseProvider(actorSystem).getLease(leaseName, leaseConfigKey, owner)

Once a lease is created you can attempt to acquire it, calling which gives you Future[Boolean] indicating whether a lease was acquired or not (it’s already acquired by another instance).

lease.acquire(leaseLostCallback).map { acquired =>

if(acquired)

// run singleton logic

else

// schedule next acquisition attempt

}.recover {

case e: Throwable => // schedule next acquisition attempt

}

If a lease gets acquired you’re free to start your singleton-based logic to do its work as it’s guaranteed to be only one instance running at that time. Otherwise you should keep polling acquire in arbitrary intervals to be ready to take over when current owner releases the lease or when it times out, making the lease available for acquisition. You can pass optional leaseLostCallback to acquire to be called when this instance suddenly loses the lease to do some cleanup etc. and go back to polling acquire . When an instance being the active singleton is about to shut down, it should explicitly release the lease rather than let it time out (not sending heartbeat in configured time). It will allow for smoother and shorter transitions. In case of current lease owner failure it’s a combination of heartbeat-interval and heartbeat-timeout configuration properties that defines how long it will take for the lease to be available for the other instances to pick it up.

That’s basically all that is to using Lease in general. Now let’s look at a Kubernetes-specific lease that we use. In general, as stated in the docs, that lease implementation can be used as an enhancement for Split Brain Resolver, Cluster Singleton and Cluster Sharding increasing consistency of these, but it can simply be used more directly as in our case — to manage simple singleton-based service deployed on Kubernetes.

It’s built on top of Kubernetes CRD (Custom Resource Definition) that already provides concurrency and consistency control so that it’s enough to build a kind of distributed lock on. This also means that lease resource type needs to be created first (together with corresponding RBAC) in the cluster. That’s a one-time operation requiring cluster admin privileges.

apiVersion: apiextensions.k8s.io/v1beta1

name: leases.akka.io

spec:

group: akka.io

version: v1

scope: Namespaced

names:

plural: leases

singular: lease

kind: Lease

shortNames:

- le

and

kind: Role

apiVersion: rbac.authorization.k8s.io/v1

metadata:

name: lease-access

rules:

- apiGroups: [“akka.io”]

resources: [“leases”]

verbs: [“get”, “create”, “update”, “list”]

--

kind: RoleBinding

apiVersion: rbac.authorization.k8s.io/v1

metadata:

name: lease-access

subjects:

- kind: User

name: system:serviceaccount:<YOUR NAMSPACE>:default

roleRef:

kind: Role

name: lease-accese

apiGroup: rbac.authorization.k8s.io

This allows applications to create and update lease instances within their namespaces later, without any special privileges required. Now just a piece of configuration on the application level where kubernetes-lease already contains the defaults under akka.lease.kubernetes config key and you’re ready to use it with generic Lease API as described above. You may probably want to tweak the configuration parameters related to heartbeats to minimize takeover time depending on your needs but in general you now have your singleton service guarded from running into data consistency issues.

Summary

Sure, using a singleton pattern has its drawbacks, but it’s a matter of choosing the right tool for the job. If you decide to use one, it’s required to understand the consequences and have strong guarantees that the singleton pattern is not violated as it may result in some hard to revert data corruption.