Adds a function and tests to get robot.txt urls