modified the crawling so that the same file will not be crawled twice