updates to use less memory so we can crawl deeper into websites.