rinzi/robotstxt — Issue #1: "Disallow: /?" rule breaks matching
Author: Smilediver
Created: 2012-02-06 01:19:01+00:00
State: open

"Disallow: /?" rule should only forbid paths starting with a '/?', but it also forbids the root path '/', because rule matching is done using regexp:

`is_allow = false if url_path.match('^' + d ) || d == '/'`

Example failing robots.txt: http://a9.com/robots.txt