Only calc ip-variance on backends