update to reflect changes to the test site
add a new default url
don't respect robots.text when testing the getCrawableUris method
remove now unused pages
don't display warnings when getting the robots.txt file

solves an issue with tests, was displaying a warning on a 404 response.
Merge remote-tracking branch 'upstream/master'

Conflicts:
	src/Spider/Filter/RobotsTxt.php
	tests/spider.getCrawlableUris.phpt
	tests/spider.spider.phpt
Merge remote-tracking branch 'upstream/master'

Conflicts:
	src/Spider/Filter/RobotsTxt.php
	tests/spider.getCrawlableUris.phpt
	tests/spider.spider.phpt
Fix regression
Merge pull request #18 from mfairchild365/master

Update tests to reflect recent changes