after testing the way of reducing the usge of memory, apply to all
crawlers.