A few updates to the crawler to make things a bit more memory efficient.