Added support for web crawlers