Checks for valid robots.txt for multile sites. Uses cucumber for running tests & reporting errors.