Web crawler with index functionality