Added option shutdownOnEmptyQueue that determines if the crawler will
shut down automatically when the queue is empty or if it will keep
running.