I don't see any point in allowing web crawlers at this point, and can't really see the use case for them even after going to full production-- the web crawlers should be hitting the actual web pages that will be served by other projects, not the API endpoints hosted by this one.