disable fit gradients when they are being handled by the auxiliary ref group