I'm not convinced. If we have 5 frames queued in renderer, we can technically stop decoding 5 frames, and still not end up late. It's only when the frame we just decoded is already late for display we actually are late.

If last frame at render stage is late, nothing is queued in renderers anymore since we will skip frames to the first frame that isn't late (there is no such thing as catching up here). This also implies we are also late at decoding stage if we just amend the lateness check there by one frame interval (or half render queue size in time). So I really think we can utterly disregard render stage for lateness detection.

I still agree that we must remember actions already taken in decoders, otherwise we are lost. Problem is that we don't get that info out of ffmpeg currenctly when dropping inside it.

Absolute times at render stage is usually better since they are not sensitive to discontinuities, but it's not really required anymore, we can switch, but it must be done by passing the clock down from player in the configure call.

Imho clock object should not care about lateness. It should just be a sync time.