modified crawling logic for handling redirects