Drift metrics relative to valid incoming depth pixels rather than valid reprojections