Kolibri — A concurrent, clusterable, open-source execution framework based on Akka, with applications in search
By: Andreas Wagenmann
Tags: Search, Akka, Scalability, Scala, Search Engines

Do you need to

execute jobs involving many data points and/or many heavier executions for a variety of use-cases and keep control and track of the status?

split larger jobs into batches and execute them concurrently over multiple cluster-nodes with build-in horizontal scaling or just on your local machine?

get out of callback hell and handling concurrent access and modification of data structures?

need job-level aggregations based on (nested) concepts that can evolve during processing on single-data-sample level?

place maximum load on another system, yet not overwhelm it?

If you find yourself saying yes to some of those, continue reading …

Kolibri Logo — (Kolibri is the german word for hummingbird)

The idea to create Kolibri (very fast, very small bird) stems from my experience in the world of user-facing, request / data-heavy services, especially within the areas of e-commerce and search. Often there is a proliferation of different toolings despite strongly overlapping requirements, such as:

Indexing data: index lots of documents with maximal throughput, and know which failed

Load-testing: generate high load and provide response statistics

Search quality estimation / optimization: evaluate on many combinations of queries and parameters in as little time as possible, and reliably aggregate results on query and/or result criteria

There has been quite some improvement of availability of libraries for parallel and concurrent processing in recent years, yet Akka remains an excellent choice for this, allowing reliable and transparent processing (especially when state-keeping is involved). The following describes an open-source processing framework I am building, based on akka-cluster and written in Scala. It goes by the name of “Kolibri” (hummingbird in german). It is opinionated in the sense that it tries to be explicit and refrain from too much implicit logic. This article shall provide a very high level overview.

The base execution logic is covered in two basic libraries:

kolibri-datatypes : datatypes utilized for processing

: datatypes utilized for processing kolibri-base: processing mechanisms utilizing the cluster capabilities which akka-cluster provides. It also brings definition of controlled processes, which allows flexible result aggregation by the use of tags on processed entities, controlled result retrievals by set expectations on the execution and batchable, streaming execution of single processing entities. One focus is on lean setup avoiding implicit assumptions. Thus no framework like Play is utilized.

The idea is to have a basic processing core and add specific use cases by additional libraries providing the job definitions. Check out the kolibri-base README for descriptions how to start up a small cluster locally and run a simple batched pi-estimation by dart-throw example.

At the time of this writing both of the basic libraries (datatypes and base) have reached status of the first alpha-release. Yet while much of functionality is already in there, it needs polishing and additional testing. This article also serves to initiate discussions around requirements and use-cases, so feel welcome to comment.

Lets start…