cgiffard/node-simplecrawler — pull request #1 by cgiffard (2012-03-08 00:19:44+00:00)
Title: Development Merge

Improved URL handling and crawing, dealing with rubbish data.

Added basic cache system. Crawler does not currently request URLs with the cache system in mind, but it will save to cache.

I've decided to merge to master, because the while the cache remains in development, the crawler has had many improvements and is more stable now - and users do not have to use the cache system.