serpentem/theoricus — pull request #18 by hems (2013-01-25 18:08:25+00:00)
Title: new crawler version is ready to go ( :

+ Avoid node-phantom crashs when crawling loads of pages
+ Fixed link parsing bugs
+ Create multiple connections at once

+ Minor syntax improvements
+ Adds some of documentation
+ Improved crawling progress feedback by writing more / friendly console log messages

TODO:

- Implementing parsing using "jsdom"
- Add simultaneous number of connections as a parameter
- Externalise parsing logic in order to make custom rules without having to deal with the crawler source ( i.e. having a coffee file with "before parse", "get links", ... ) so other solutions different from "rendering full page" could be quickly implemented and versioned together with a project

Tested a few times on the current website am developing, 10 connections at once is quite fast, no crashs so far.