- Obeys "User-Agent: *" disallows
- Matches robots.txt urls with regex instead of strings