Added robots.txt support to get_hrefs so that you can be a good little
web crawler
Fixed accedental deletion of targz and fixed robots in fetch issue