edx/edx-platform — pull request #652 by ichuang (2013-08-12 20:34:15+00:00)
Title: Enable mongodb to be used as back-end for git-based authoring workflow

MIT authors have largely preferred to use the "github+LMS" workflow instead of Studio. In the github+LMS workflow, course authors do their work on a git repository; when changes are pushed, a webhook (on github) automatically triggers the LMS to reload XML course content from github, via a "gitreload".

This workflow has enabled MIT authors to employ many advanced features which are inaccessible from Studio. It also allows large course teams, with many authors, to avoid stepping on each other, and to take advantage of social coding and network effects arising from using github. It also lets course teams keep track of multiple revisions, to revert mistaken changes, and to use one repository for multiple course branches. It also allows researchers to track the entire history of course content changes, eg so that if mistakes in a midterm exam are corrected, this is recorded in the git repo logs, and can be reconciled with records of student responses.

The downside has been that loading XML files into the bare, non-mongo LMS is painful. Each thread of an LMS must load all the XML into memory. This causes the LMS to take a long time to restart, when many courses are loaded. It can also lead to inconsistencies between LMS responses, when the threads of an LMS hold different versions of XML content in memory.

The solution to this is to replace the memory-backed LMS with a mongo-backed LMS, in the github based workflow. This new "git+edge" workflow, or "git+mongoLMS" workflow, then has all the benefits of github's social coding, together with the benefits of using a central database backend for the LMS instances.

This solution has been in discussion since the spring of 2013 (re email with @ormsbee), and many of the needed pieces have been put in place since then. The last remaining issue has been the fact that certain content, ie script code stored in libraries, and static content stored hierarchically, get lost or loose hierarchy, when imported into mongo. Also, when a course has a large amount of static content, it is very inefficient to reload all of this into the mongodb each time an author makes a minor change (eg 8.02x, which has many GB of static content associated with its TEAL simulations).

This PR puts in a last piece needed to make the "git+mongoLMS" workflow possible and functional. It does this by circumventing the database for static and code content. Specifically, it adds the ability to import course content (into CMS / edge) without static content, and without rewriting static links. It changes xml_importer.py and import.py to introduce `static_asset_path` as an LMS namespace metadata variable, which is used to determine if static links should be rewritten (as `c4x://...`) or left alone. The import command now takes a `--nostatic` flag on the command line, to indicate when this should be done.

With this change, and by configuring nginx to look in course /static directories for individual course static content, the "git+mongoLMS" workflow is enabled. To enable code to work, course repo directories are symlinked to the course number directories

This PR is now in use in the production residential MITx systems at MIT.

No tests are provided -- pending discussion and comments.