Queens-Hacks/qcumber-scraper — pull request #2 by mystor (2013-12-19 06:58:28+00:00)
Title: Threaded Scraper take 1

I took a quick shot at making a threaded scraper. Unfortunately, I was far too optimistic about how independent each server request is (There is too much server-side state :cry:)

So, the code on this branch doesn't completely work. I have functioning login, and requests. I will be refactoring the code over the next few days, and it should be able to scrape SOLUS relatively easily (I will probably do the scraping by-letter, with a separate session for each one. I *THINK* that that should work).

I have been testing with 10 concurrent request threads, and SOLUS doesn't seem to mind at all. So I will probably stick with that number

Also, about the commit messages, I'm sorry.