edx/edx-analytics-pipeline — pull request #31 by brianhw (2014-10-22 03:36:34+00:00)
Title: Support validation of enrollment.

Includes:

* Synthesize validation events from Sqoop dumps of student_courseenrollment.

* Compare validation events with enroll/unenroll events. Includes unit tests.
Supports additional flags for:
* Outputting tuples (for debugging) or synthetic events (for production)
* Ignoring any events before the first validation event (for each user/course pair).

The latter permits checking for errors in an incremental manner, by using an interval to specify a tighter period of time (e.g. a day or a few days) which includes a set of enrollment validation events. The run can then test the consistency of events since the enrollment validation, up to the end of the interval (which selects all events included). This could be set up to run on a daily basis, to help flag inconsistencies more quickly as well as to generate new synthetic events in an incremental manner.