add a custom downloader which will only download html documents
use the new downloader
add a basic scan script which will only report URLs and total page count
fix a regression

respect robots.txt