Refs #18 Added utility class and logic in RobotsTxtServer and
PageFetcher