VAAPI: Remove rubbish from forward and backwardRefs