handle bad URLs in robots