Bugfixes and robots.txt parser enhacements.

- Obeying robots.txt can now be toggled.
- robots.txt files are fetched for subdomains.
- the crawl-delay directive is now parsed.