gpolitis/node-tarantula — pull request #7 by robcolburn (2014-01-23 22:55:56+00:00)
Title: Queues

This involves some pretty big changes to the interface.

1. Firstly, there is thread concurrency through on async's queues. We can pass in the max amount of concurrency, which I've named "legs" to keep with our fun alias'ing scheme. I believe this negates the need for "politeness", but we could add that back.
2. There are a lot more events to listen to. You can use the example-wikipedia to preview them all. This helps us push our concerns about console off to the implementor.
3. There is an option to "stayInRange" which will keep us on a single domain (common use-case, I believe). It's a bit more extensible than a "domain", because use can pass in a url with a path, or a series of url's with path. Then it won't leave that folder range.

Next steps:
* I'd like to mod code the a bit more OO. Putting the core methods on Tarantula's prototype, and letting implementors override them. This will change the interface a bit more, but it will help alleviate the deep nesting, and result in concise functions.
* I'd like to add an optional PhantomJS backend. This is actually my core motivation. Other PhantomJS-based crawlers have somewhat painful interfaces.