only respond to crawlers where the domain has been whitelisted (prod)