serpentem/theoricus — pull request #17 by hems (2013-01-25 17:37:46+00:00)
Title: new crawler version is ready to go ( :

~> Unfortunately so far i had no time to implement tests or write a WIKI, but should be pretty straight forward ( CLI API still the same )

+ Avoid node-phantom crashs when crawling loads of pages
+ Fixed link parsing bugs ( should be improved in the future using jsdom )
+ Create multiple connections at once

+ Minor syntax improvements
+ Adds some of documentation
+ Improved crawling progress feedback by writing more / friendly console log messages

Tested a few times on the current website am developing, 10 connections at once is quite fast, no crashs so far.