added a probably temporary and imprefect robots.txt sitemap reading system, added statsd statistics reporting with the flag --statsd, moved sleeping for crawl delay to do_jobs, instead of making the task yielder do it (this was done to allow for the possibility of the yielder putting in to a queue instead of yielding.

Signed-off-by: mep <miethpo@gmail.com>