add SCHEDULER_IDLE_BEFORE_CLOSE to prevent the spider from being closed when distributed crawling