But if it is not due, then it blocks, and that's what hurts performance (we stop submitting video packets to GPU).
Personally I think this code should be in a separate thread from video decode, but that's a bigger change.