gpolitis/node-tarantula — pull request #8 by robcolburn (2014-01-31 23:58:26+00:00)
Title: Massive Refactor for PhanomJS Support

So, this is my massive refactor for PhantomJS support. It borrows concepts bunch from other Node/Phantom modules out there.

The largest refactoring comes from abstracting out the actual request/parsing to their own file. This allows the implementor to swap out the underling engine. We could theoretically do this on the fly too… Passing out different types of requests to different clients, or building out a command line api.

The good…
* PhantomJS web crawler out of the box, ready to go!
* Support for UserAgent and HTTP Proxies

The bad…
* The documentation needs some coming through, a lot of docs aren't written or are out of date.
* There is a list of User Agents, but there a bit old, and anything but comprehensive.

The ugly…
* Cheerio's incomplete DOM gets annoying, it may be nice to add a parsing client with an accurate DOM underneath. Again, shouldn't be too hard, copy/paste the Cheerio implementation, and update where needed.
* There is a bunch of features from Request and/or Phantom, that are specific. They are hidden by the abstraction layer. Perhaps they could be better exposed to the implementation side.

Future Ideas…
* Stubs ready for build-your-own test framework
* HTTP-Auth'd proxies - should be easy

At this point, I've met all of my immediate needs for the module. But, I think is now one of the best crawler modules on npm. It's been fun.

Once this is in, I'm going to release an implementation module with command line options.

Apologies, it's a 1k line PR. :D