update to reflect changes to the test site
add a new default url
don't respect robots.text when testing the getCrawableUris method
remove now unused pages
don't display warnings when getting the robots.txt file

solves an issue with tests, was displaying a warning on a 404 response.