changed implementation of crawl_web