add links for web crawlers