hopefully better robots.txt processing, hopefully without false positives