mperham/sidekiq — pull request #1603 by mperham (2014-03-25 18:57:23+00:00)
Title: Client redis sharding

Allow the Sidekiq client API to push to an arbitrary connection pool so Sidekiq can shard across many different Redis server instances. This should allow arbitrary scale up, well beyond our current limits (about 8000 jobs/sec). Very simple possible API:

```ruby
SHARD_A = ConnectionPool.new { Redis.connect }
SHARD_B = ConnectionPool.new { Redis.connect }

class HardWorker
include Sidekiq::Worker
# Shard based on type. All HardWorker jobs
# will go to SHARD_A by default.
sidekiq_options :pool => SHARD_A
def perform(count)
end
end

# Should go to SHARD_A
HardWorker.perform_async(5)

# Shard based on some external function, e.g. the
# state of residence of the user.
# ALL jobs within the "to" block will go to
# SHARD_B.
Sidekiq::Worker.to(SHARD_B) do
HardWorker.perform_async(5)
HardWorker.perform_async(4)
end
```

It goes without saying that this can be complicated very quickly. Don't shard unless you absolutely need to. An alternative to sharding is to split up your application into smaller applications, each with their own Sidekiq jobs and Redis instance.