LearnBoost/socket.io — Issue #686: issues scaling across processes with the Redis store
Author: brettkiefer
Created: 2011-12-15 20:13:45+00:00
State: open

First, I'd like to say thank you to the Socket.io devs for the great work, and for providing a way to scale this across processes with the Redis store! I think that, this is a really promising thing, even though we've been encountering a few issues.

We've been having some problems scaling up socket.io with the Redis store. The problem appears to be that the PubSub is too chatty. For instance, each time a socket disconnects, every other process subscribes to the dispatch:[id] channel for that socket and waits for the close message, then unsubscribes from that channel. So in the case where you have to restart your entire service (let's say on a non-backward-compatible release), you have a TOTAL number of sockets connected to all of your worker processes (let's call that n) and a number of processes (let's call that p) and each process has to subscribe and unsubscribe for each socket, and the complexity of a subscribe in Redis is O(n) where n is the number of already-subscribed listeners on a channel. So that would seem to be O(n*p^2) work for Redis, and anyway is at least O(n) for every worker process.

Since we are only targeting websockets, we are working around this by disabling all pubsub channels except handshake, and only doing the initial handshake broadcast (so all processes other than the one that completes the handshake simply discard the data from the 'handshaken' hash as though the handshake was never finished). This appears to mitigate the issues sufficiently that we can run websockets via socket.io in production across our 36 node.js processes on 3 machines, but it still obviously won't scale indefinitely because every process still has to process a handshake message on the 'handshake' channel for each handshake initiated.

I propose that this would scale across processes if it used Redis as a store rather than publishing all data to all workers in these cases. For instance, rather than publishing on 'handshake', write a handshake:[id] key in Redis with a timeout of 30s and then let the process that completes the handshake update that key (or delete it and write a handshake-completed:[id] key).

I understand that this gets more complicated for other transports, where buffering responses becomes important, but it seems like this might be better-solved by storing the buffer as a capped list of JSON objects in Redis (our AJAX short-polling fallback uses that approace, and it works well), rather than using pubsub to pass them around.

I apologize for the lengthy and unsolicited suggestion, and we'd love to help if y'all are interested in going this direction. Also, if we're just doing something in a completely wrong or unanticipated way, please let me know. Thanks!