Added sitemap crawler function and handled javascript-only a tags