enable WHOIS lookups, save as text file. tweak robots_txt_allow? to check for Allow rules.