Improved robots.txt handling: when a wildcard rule disallows all and a
specific rule allows a user-agent, the wildcard rule was applied
nonetheless.

The change is to maintain two sets: one of generic directives and one
specific for the user agent. If either of the two allows a page, it is
accepted.