iParadigms/walker — pull request #109 by peterdietz (2014-12-15 17:08:16+00:00)
Title: Fetcher robots.Test needs URL.RequestURI, not string(URL)

I've noticed that crawls have been including tons more links than they ought, like fetching search/browse results that a robots.txt explicitly excludes. I've discovered that although walker is indeed doing the right thing by checking robots.txt, and checking links, the issue is that robots.Test( ) wants url.RequestURI(), not the full url.

Sadly they aren't documenting their function...
https://github.com/temoto/robotstxt-go/blob/2f1297117982b399076203fc13b76a79796a881e/robotstxt.go#L193
`func (g *Group) Test(path string) bool {`

But this test is indicating that it doesn't want the domain, just the path, and so RequestURI includes path and urlparams.
https://github.com/temoto/robotstxt-go/blob/2f1297117982b399076203fc13b76a79796a881e/robotstxt_test.go#L230
`if !group.Test("/") {`

I didn't know how to properly test fetcher.go's fetchAndHandle, would you have to build ton's of mocks? So, I've added robot_test.go, which tests robots.Test on this https://github.com/temoto/robotstxt-go library. For a more visual test run `go test test/robottestmain.go`, which output's:

```
urlSample: https://www.google.com/search?q=walker
Fetching robots.txt: https://www.google.com/robots.txt
TRUE means allow to fetch, FALSE means prohibited by robots.txt
Full: https://www.google.com/search?q=walker : true
RequestURI: /search?q=walker : false
Full: https://www.google.com/about/ : true
RequestURI: /about/ : true
```

Google's robots.txt disallow's /search, but doesn't disallow /about.