Fix to avoid serializing the priority queue in the common case