diaspora/diaspora — pull request #4769 by Ruxton (2014-02-09 12:25:34+00:00)
Title: Updating queries for the stream to be faster

Tested with masses of users, this one change results in a massive performance increase.

* Prior to accepting for merge I'll revert those logs.

I had a user in my dev environment, I gave him 30,000 followers, like poor DiasporaHQ probably has. His stream was taking quite a while to load. it now takes about 1 second. Both the below messages are from the dev log on my quad core laptop, performed after restarting the server, hitting the stream once to warm up the app, then refreshing once more. They're taken from the JSON fetch of the stream.

The next place I'll tackle is posting as the user with 30,000. The things that happen after this tie up the server process for minutes (there's quite a lot of "ID IN [*30,00 IDS GO HERE*]".

```
Started GET "/stream?_=1391948397159" for 127.0.0.1 at 2014-02-09 20:19:57 +0800
Processing by StreamsController#multi as JSON
```

Before:
```
Completed 200 OK in 34196.1ms (Views: 279.6ms | ActiveRecord: 33505.6ms)
```

After:
```
Completed 200 OK in 1361.3ms (Views: 235.7ms | ActiveRecord: 1105.7ms)
```

Queries that do things like "ID IN [array goes here]" are super expensive, the longer that list gets, the longer the query takes. Sub-selects are slowish too, but nowhere near as bad as that.