Include a feature for monitoring robots.txt files.