Updated logic for crawling