Updates from past 10 days
error handling, queueing

Improved error handling a bit, Updated domain queue to  queue hosts as
opposed to full on urls, updated robotstxt to evaluate a crawldirective
if a value for the directive exists