Added missing support for priority and dont_filter to request pickling support
Allow for skipping the start URLs - this is for worker processes that get fed from a queue.
Optionally wait around when spider is idle.