Wikia/data-science-toolkit — pull request #6 by tristaneuan (2014-05-20 20:10:07+00:00)
Title: Page LDA, recommendation XLS, logging, loadbalancing, parsing & data-ex changes

There's a lot of stuff in here...
Wrote page LDA client and server.
Added script to write recommendation XLS from CSV.
Moved log() definition from wikia_dstk/lda/init.py to wikia_dstk/init.py and corrected relative import paths everywhere it's used.
All logs write to /var/log now.
Modified loadbalancing to ensure there are enough available spots for instances and spot instance requests before launching; waits and retries if not.
Some performance improvements in the parsing and data extraction pipeline with multiprocessing.
Split wikia_dstk/pipeline/monitor.py into two separate scripts for parsing and document-level data extraction. These monitor scripts include user_data scripts to be passed to EC2 instances in order to ensure we're using the latest versions of our libraries.
Updated parser to use data-science-toolkit instead of nlp-rest-client. Deprecated service/daemon system, and parser poller/daemon are launched via user_data.
I didn't touch anything in the classification subdirectory; not sure why diffs for these are showing up green.