update integration tests to be a bit more specific
clear webmocks after use and add tests and functionality for following robots.txt