fixed some minor bugs, check for search engines in is_known_crawler?