sakaiproject/Hilary — pull request #312 by mrvisser (2013-01-07 15:31:54+00:00)
Title: Activity with aggregation

This PR implements most of activities with aggregation. Still need to write unit tests, hook up thumbnails and fine-tune little things like including comment context in the activity entities.

I backed off delivering activities with such aggressive speed as before. The previous activities branch w/o aggregation would deliver activities immediately when routed. This implementation queues them into "buckets" where they can be processed.

The reason they are queued is because there are a fair amount of race conditions involved if you have multiple processes delivering and aggregating activities for the same routes and types. You can end up with accidentally duplicating aggregates in a user's feed, and also delivering multiple single-activities intermittently when they should have been aggregated.

The bucketed approach ensures that activities of the same route and type are queued in the same bucket, which means that they will be collected in serial by a single process, which eliminates all the race conditions. The reason there are multiple buckets instead of a single queue is because, since the point is to ensure only one process can collect the queue at one time, a single queue would mean activity collection cannot scale at all beyond one server. Since multiple buckets can be collected concurrently, we can increase the number of buckets to reflect the number of processes we have collecting activities, giving us the ability to scale activity performance with more hardware.