franzenzenhofer/robotstxt — Issue #3: ? at the end of the url does not get tested
Author: franzenzenhofer
Created: 2011-10-16 19:53:18+00:00
State: open

is:

/test?

gets tested as

/test

should:

/test?

gets tested as

/test?

probably some bug with how we handle parsed URIs (as ? URIs does not make a lot of sense in the real world, but are a common robots.txt tes scenario)