improving mapping to googlebot and disallowing craling on /backend/crawler