webcrawler functionality