Validate urls against robots.txt rules.