Added test to confirm incorrect use of robots.Test
Merge pull request #110 from iParadigms/bad-robots-test

Added test to confirm incorrect use of robots.Test
Fix bug in fetcher, as robots.Test require the requestURI
Allow TestBasicRobotsDisallow to run, since code properly checks robots
Merge pull request #109 from LongsightGroup/robots-requesturi

Fetcher robots.Test needs URL.RequestURI, not string(URL)
Merge branch 'master' into TRN-126