Crawler appears to be working properly now. No dups, loop appears to finish