added the process_spider_exception method in middleware to release lock and schedule next request when exception occurs