fixed workers dieing when the broker has no jobs to give them, fixed a site's crawl delay being None instead of the default when a worker fails to get the site's robots.txt.

Signed-off-by: mep <miethpo@gmail.com>