Demuxer: Don't workaround in demuxer

Follow up of #5722