calling crawler with robots + changes in validatr

added code where the crawlers fail because of validator checks