added throttling and smarter crawling