Fixed the issue when spider is prematurely put into idle state, even
though its errback() founction schedules more requests.