edx/cs_comments_service — pull request #51 by jimabramson (2013-09-10 13:45:01+00:00)
Title: DO NOT MERGE removing recursive queries in thread view

@gwprice @kevinchugh @e0d

This change set addresses performance problems with GET /api/v1/threads/:thread_id?recursive=True. Previously this view was built by recursing over the contents collection in mongo document by document, which degrades exponentially as the number of comments increases.

Here we add a sparse sequence of scalar sorting keys ("sk.p0", "sk.p1", "sk.p2") to each new comment document at create_time, and an accompanying index (which also includes "created_at"). The sort key values are copied from the "parent_ids" array automatically populated by Mongoid::Tree, except that when a comment has no parent, its own id is provided as the first sort key. This allows access to an entire thread's content, sorted in display order, using a single indexed query. NOTE: only three levels of comment are presently supported; this is one more level than is presently allowed by edX Forums.

Also, previously each individual comment load (call to Comment.to_hash) involved a lookup for the author's username, potentially incurring another mongo query each time. This change copies the author's username into the comment document, making the lookup unnecessary.

I'm seeing very promising performance improvement running locally against a clone of prod, next step is a proper loadtest in a fully-inflated environment.