Added tests to disallowed directories for robots.txt