Initial Commit
Computer transfer
Indexers now gracefully die on request, after indexing there current page, rescue from 'Encoding::CompatibilityError' and mark as crawled to prevent spamming url.
Fixed duplicate '/' when adding url to db
Added global default crawl-delay
Merge https://github.com/cyberarm/royageur