Database of robots.txt-derived permissions