App testing the existence of the disallowed paths on a robots.txt file