Proposal to adopt RabbitMQ

# Proposal to adopt RabbitMQ

I would like to propose a solution for the longer term.  That is, just one solution option that _could_ be implemented in the timeframe for which the nearer term, interim, solution options are intended to buy us a longer runway, such as [application-scope sharding](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7199).

This proposal's subject is the current status quo application use of the sidekiq framework for asynchronous task processing with redis cache as a FIFO queue/locking mechanic.

As a software developer for many years, I have experienced the consequences of inter-system and inter-service bottlenecks which are often introduced by misuse of software patterns and frameworks.  Of course intentions are always good and it is almost always the case that some software pattern implementations tests well and work well in the nearer term, at small scales, or with simplistic/naive test scenarios.  However, as we systems engineers know all too well, as an application's usage profile grows, or adherence to application software development best practices drifts in only one or two code modules, feature releases, or commits, things become more complicated.  Sometimes software frameworks and patterns are employed with default configurations that are seldom if ever adjusted to suit the application usage.  Sometimes such solutions are simply not designed to be used feasibly at certain scales.

## RabbitMQ or sidekiq?

Approaches to clustering Redis include single node multi-process distributed request proxy load balancing, multi-node single-process distributed request load balancing, and also multi-node multi-process-per-node distributed request load balancing.  As far as I know, we are using none of these, except maybe multi-node single-process distributed request load balancing, but even then, the additional nodes are not being used as clustered worker nodes to perform load-sharing, but as failover nodes with replicated data. Furthermore, it seems unlikely that all read requests are getting properly distributed exclusively to the failover nodes, but instead are being equally distributed to all three nodes, which overburdens the master Redis node that should be solely responsible for write requests. (https://redis.io/topics/faq, https://redis.io/topics/partitioning, https://redislabs.com/redis-enterprise/technology/redis-enterprise-cluster-architecture/)

On the other hand, adopting a write-only usage for the master node could cause failures with a variety of application usages such as any update-on-miss (`SETNX`) operations. (https://redis.io/commands/setnx, https://redis.io/commands/setnx#handling-deadlocks)

Such concerns are further complicated by the best practice patterns for using the sidekiq framework itself.  Configuring task prioritization with sidekiq can be troublesome.  For instance, Phil Sturgeon points out that, "defining multiple queues in your Sidekiq config does not distribute work evenly between them." [Nov 16 2016](https://phil.tech/2016/11/16/tips-on-sidekiq-queues/)

With RabbitMQ working as a broker for sending parametric info for tasks with appropriate background-processing/asynchronous work profiles, and distributing messages to categorical pools of asynchronous workers, cpu load is automatically balanced between all cores by design.  RabbitMQ's multi-threaded Erlang platform is built for services requiring concurrency and makes horizontal scaling very simple from a configuration and deployment perspective.

This is not the case with Redis, which is built to be a single-threaded single process application by design.  This design is adequate for its commonly marketed use cases, but can make scaling it horizontally somewhat complicated, as explained above.

## Support for microservice architecture

Message broker technologies like RabbitMQ and Kafka can also support an architectural transition to microservices, which is where I understand GitLab may be headed towards in the future.  Martin Fowler had plenty to say about [Microservice Trade-Offs](https://martinfowler.com/articles/microservice-trade-offs.html) in 2015.  Another good resource for reading more about distributed services communication is here over at [microservices.io](https://microservices.io/patterns/communication-style/messaging.html) which also has coverage on other conventional communication techniques like [RPI](https://microservices.io/patterns/communication-style/rpi.html).

The take-away here is that microservice architectural patterns are commonly accompanied by message broker technologies.  Technologies like sidekiq certainly have their place, but distributed inter-service communication orchestration might not be that place.  So, if a more distributed architecture is in the stars for GitLab, then perhaps we can take this opportunity to begin to lay the groundwork for such a transition now.

My intention here is to cultivate discussion around the singular problem presented by sidekiq which appears to be that horizontal scaling with sidekiq and redis is more challenging compared to other options like RabbitMQ.


## RabbitMQ and Sneakers

Sneakers is a ruby background job processing framework that uses RabbitMQ.

> "Nor could I allow the single-point-of-failure that being Redis, which isn't really suitable for a highly-available background processing framework - I couldn't lose messages (it's worth mentioning that Redis is often my go-to swiss-army knife and you'll have to pry it from my dying corpse)."

-- [Dotan J. Nahum](https://github.com/jondot/sneakers/wiki/Why-i-built-it#cruby-performance-and-high-availability), Jan 10, 2014, https://github.com/jondot/sneakers

Additional advantages of using Sneakers:

> Compared to Sidekiq, per my use case, I needed

>     A great-performing framework limited only by broker speed - at least 1000req/s acknowledged and persisted on EC2-Large (Sneakers does more than that), and
>     *That would use all cores*
>     A highly available processing framework (here we have same guarantees as RabbitMQ offers, which is great)
>     A familiar DSL/API that also supports advanced messaging semantics such as reject, requeue, acknowledge, etc, and
>     That would not expose the whole guts of AMQP at me, but just-enough from it.

> And irrelevant of the comparison to Sidekiq or any other background processing framework I needed

>     It should use a ruby that doesn't care about content of gems and can run C-extensions. MRI.
>     A production-ready package that holds all of these together allowing me to be as lazy as possible
>     Metrics and logging baked in
>     Convenient deployment, maintenance and supervision story

-- [Dotan J. Nahum](https://github.com/jondot/sneakers/wiki/Why-i-built-it#sneakers)

Additional reading:

* [Messages on Rails Part 3: RabbitMQ](https://karolgalanciak.com/blog/2019/06/23/messages-on-rails-part-3-rabbitmq/)
* [RabbitMQ Scheduling Messages with Sneakers](https://medium.com/@twobuckchuck/rabbitmq-scheduling-messages-with-sneakers-18089e8aa7d2)

## Additional sharding and prioritization using queues

Solutions like RabbitMQ provide support for application features like programmatic sharding for job criticality levels and domains by using RabbitMQ first-class queues and channels, instead of redis locking patterns which are contentious and susceptible to inconsistency errors during fail-overs.

> "Another core concept of job framework is queues. A typical app would have a dozen queues (critical, default, webhooks, low, imports, payments etc) and the developer would have to choose one for their job. As you can see, the set of queues has a mix of priority based queues (critical, default, low) and domain-specific queues (webhooks, imports, payments)."

-- [Kir Shatrov](https://kirshatrov.com/2019/01/03/state-of-background-jobs/), 03 Jan 2019

Additional reading: [The State of Background Jobs in 2019](https://kirshatrov.com/2019/01/03/state-of-background-jobs/)

## Summary

To summarize, my limited research and experience informs me that it can be a complicated design problem to scale redis when relying on lock acquisition mechanics for asynchronous job queueing.  It requires a sophisticated coordination between good architectural deployment patterns and programmatic adherence to certain application software patterns in order to ensure both reliability, but also horizontal scalability.

Personally, I suspect that GitLab has outgrown redis-backed sidekiq, to an extent.  I recommend incorporating RabbitMQ and one of the many available Ruby libraries and frameworks, such as Sneakers or Bunny, into the GitLab Rails deployment configuration and make the necessary changes to the codebase in order to interface with appropriately considered queues for invoking categorical asynchronous tasks, and reserve redis use for the purposes of caching simplistic session token and frequently read/infrequently modified application data.

## Falsifiability

My suspicions could certainly be mistaken.  I have not yet grokked the numbers involved in the profiles of our redis usage by our application.  It is currently difficult to do such a thing because our present usage is somewhat functionally overloading one of the redis clusters.  This will probably be remedied by [this issue](https://gitlab.com/gitlab-com/gl-infra/infrastructure/issues/7199), and we'll soon have a clearer picture of usage which is dominating the bandwidth of the existing redis system resources.  It may be that once splitting redis sidekiq from redis persistent has completed, and obvious mal-patterns illuminated and eliminated, there will be plenty of room to grow, and the use-case for a message broker service obviated.

Related issues: https://gitlab.com/gitlab-com/gl-infra/production/issues/928

Tagging team members for visibility and feedback solicitation: @glopezfernandez, @stanhu, @andrewn, @ahanselka, @msmiley, @craig, @craigf, @cmiskell, @devin, @skarbek, @jarv, @ansdval, @Finotto

Story points: 1.0