RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [200, 2]], which is output 0 of AsStridedBackward0, is at version 1401; expected version 1400 instead

Lukas_Scholz: <p>Hi there!<br>
When trying to do a backward-pass in my model I receive the following error:</p>
<p>Things I already tried:</p>
<ul>
<li><code>inplace=True</code> for <code>Relu()</code></li>
<li><code>clone()</code> on Tensors in <code>GaussianMLP</code></li>
<li>trying to replace all python inplace operations</li>
</ul>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> I see that you helped a lot on other issues like this. Am I missing here something?</p>
<pre><code class="lang-auto">z_vec, x_vec, uncertainty_vec, epistemic_vec, aleatoric_vec, cost_vec, dist_vec = CLUE_explainer.optimise(
        min_steps=3, max_steps=35,
        n_early_stop=3)
</code></pre>
<p>clue.py</p>
<pre><code class="lang-auto">torch.autograd.set_detect_anomaly(True)

class CLUE(BaseNet):
    def __init__(self, VAE, BNN, original_x, uncertainty_weight, aleatoric_weight, epistemic_weight, prior_weight, distance_weight,
                 latent_L2_weight, prediction_similarity_weight,
                 lr, desired_preds=None, cond_mask=None, distance_metric=None, z_init=None, norm_MNIST=False, flatten_BNN=False,
                 regression=False, prob_BNN=True, cuda=True):

        # remove constructor for minimalizm
        self.original_x = torch.Tensor(original_x)

        self.prob_BNN = prob_BNN
        self.cuda = cuda
        if self.cuda:
            self.original_x = self.original_x.cuda()
            if self.desired_preds is not None:
                self.desired_preds = self.desired_preds.cuda()
        self.cond_mask = cond_mask

        self.trainable_params = list()

        if self.VAE is None:
            self.trainable_params.append(nn.Parameter(original_x))
        else:
            self.z_dim = VAE.latent_dim
            if z_init is not None:
                self.z_init = torch.Tensor(z_init)
                if cuda:
                    self.z_init = self.z_init.cuda()
                self.z = nn.Parameter(self.z_init)
                self.trainable_params.append(self.z)
            else:
                self.z_init = torch.zeros(self.z_dim).unsqueeze(0).repeat(original_x.shape[0], 1)
                if self.cuda:
                    self.z_init = self.z_init.cuda()
                self.z = nn.Parameter(self.z_init)
                self.trainable_params.append(self.z)

        self.optimizer = Adam(self.trainable_params, lr=lr)

    def randomise_z_init(self, std):
        eps = torch.randn(self.z.shape).type(self.z.type())
        self.z.data = std * eps + self.z_init
        return None

    def pred_dist(self, preds):
        assert self.desired_preds is not None

        if self.regression:
            dist = F.mse_loss(preds, self.desired_preds, reduction='none')
        else:

            if len(self.desired_preds.shape) == 1 or self.desired_preds.shape[1] == 1:
                dist = F.nll_loss(preds, self.desired_preds, reduction='none')
            else:
                dist = -(torch.log(preds) * self.desired_preds).sum(dim=1)
        return dist

    def uncertainty_from_z(self):
        x = self.VAE.regenerate(self.z, grad=True)

        if self.flatten_BNN:
            to_BNN = x.view(x.shape[0], -1)
        else:
            to_BNN = x

        if self.norm_MNIST:
            to_BNN = calculate_mnist_mean_std_norm(to_BNN)

        if self.prob_BNN:
            if self.regression:
                mu_vec, std_vec = self.BNN.sample_predict(to_BNN, num_samples=0, grad=True)
                total_uncertainty, aleatoric_uncertainty, epistemic_uncertainty = decompose_std_gauss(mu_vec, std_vec)
                preds = mu_vec.mean(dim=0)
            else:
                probs = self.BNN.sample_predict(to_BNN, num_samples=0, grad=True)
                total_uncertainty, aleatoric_uncertainty, epistemic_uncertainty = decompose_entropy_cat(probs)
                preds = probs.mean(dim=0)
        else:
            if self.regression:
                mu, std = self.BNN.predict(to_BNN, grad=True)
                total_uncertainty = std.squeeze(1)
                aleatoric_uncertainty = total_uncertainty
                epistemic_uncertainty = total_uncertainty * 0
                preds = mu
            else:
                probs = self.BNN.predict(to_BNN, grad=True)
                total_uncertainty = -(probs * torch.log(probs + 1e-10)).sum(dim=1, keepdim=False)
                aleatoric_uncertainty = total_uncertainty
                epistemic_uncertainty = total_uncertainty * 0
                preds = probs

        return total_uncertainty, aleatoric_uncertainty, epistemic_uncertainty, x, preds

    def get_objective(self, x, total_uncertainty, aleatoric_uncertainty, epistemic_uncertainty, preds):
        objective = self.uncertainty_weight * total_uncertainty + self.aleatoric_weight * aleatoric_uncertainty + \
                    self.epistemic_weight * epistemic_uncertainty

        if self.VAE is not None and self.cond_mask is None and self.prior_weight &gt; 0:
            try:
                prior_loglike = self.VAE.prior.log_prob(self.z).sum(dim=1)
            except:
                prior_loglike = self.VAEAC.get_prior(self.original_x, self.cond_mask, flatten=False).log_prob(self.z).sum(dim=1)
            objective = objective + self.prior_weight * prior_loglike

        if self.latent_L2_weight != 0 and self.latent_L2_weight is not None:
            latent_dist = F.mse_loss(self.z, self.z_init, reduction='none').view(x.shape[0], -1).sum(dim=1)
            objective = objective + self.latent_L2_weight * latent_dist

        if self.desired_preds is not None:
            pred_dist = self.pred_dist(preds).view(preds.shape[0], -1).sum(dim=1)
            objective = objective + self.prediction_similarity_weight * pred_dist

        if self.distance_metric is not None:
            dist = self.distance_metric(x, self.original_x).view(x.shape[0], -1).sum(dim=1)
            objective = objective + self.distance_weight * dist

            return objective, self.distance_weight * dist
        else:
            return objective, 0

    def optimise(self, min_steps=3, max_steps=25,
                 n_early_stop=3):
        z_vec = [self.z.data.cpu().numpy()]
        x_vec = []
        uncertainty_vec = np.zeros((max_steps, self.z.shape[0]))
        aleatoric_vec = np.zeros((max_steps, self.z.shape[0]))
        epistemic_vec = np.zeros((max_steps, self.z.shape[0]))
        dist_vec = np.zeros((max_steps, self.z.shape[0]))
        cost_vec = np.zeros((max_steps, self.z.shape[0])) 

        it_mask = np.zeros(self.z.shape[0])

        for step_idx in range(max_steps):
            self.optimizer.zero_grad()
            total_uncertainty, aleatoric_uncertainty, epistemic_uncertainty, x, preds = self.uncertainty_from_z()
            objective, w_dist = self.get_objective(x, total_uncertainty, aleatoric_uncertainty, epistemic_uncertainty, preds)

            objective.sum(dim=0).backward()

            self.optimizer.step()

            uncertainty_vec[step_idx, :] = total_uncertainty.data.cpu().numpy()
            aleatoric_vec[step_idx, :] = aleatoric_uncertainty.data.cpu().numpy()
            epistemic_vec[step_idx, :] = epistemic_uncertainty.data.cpu().numpy()
            dist_vec[step_idx, :] = (w_dist.data.cpu().numpy())
            cost_vec[step_idx, :] = (objective.data.cpu().numpy())
            x_vec.append(x.detach().cpu().numpy()) 
            z_vec.append(self.z.detach().cpu().numpy())

            it_mask = CLUE.update_stopvec(cost_vec, it_mask, step_idx, n_early_stop, min_steps)

        x = self.VAE.regenerate(self.z, grad=False).data
        x_vec.append(x)
        x_vec = [i.cpu().numpy() for i in x_vec]
        x_vec = np.stack(x_vec)
        z_vec = np.stack(z_vec)

        uncertainty_vec, epistemic_vec, aleatoric_vec, dist_vec, cost_vec, z_vec, x_vec = CLUE.apply_stopvec(it_mask,
                                                                                                             uncertainty_vec, epistemic_vec,
                                                                                                             aleatoric_vec, dist_vec, cost_vec, z_vec,
                                                                                                             x_vec,
                                                                                                             n_early_stop)
        return z_vec, x_vec, uncertainty_vec, epistemic_vec, aleatoric_vec, cost_vec, dist_vec

    @staticmethod
    def update_stopvec(cost_vec, it_mask, step_idx, n_early_stop, min_steps):
        asymptotic_rel = np.abs(cost_vec[step_idx - n_early_stop, :] - cost_vec[step_idx, :]) &lt; cost_vec[0, :] * 1e-2
        asymptotic_abs = np.abs(cost_vec[step_idx - n_early_stop, :] - cost_vec[step_idx, :]) &lt; 1e-3

        if step_idx &gt; min_steps:
            condition_sum = asymptotic_rel + asymptotic_abs
        else:
            condition_sum = np.array([0])

        stop_vec = condition_sum.clip(max=1, min=0)

        to_mask = (it_mask == 0).astype(int) * stop_vec
        it_mask[to_mask == 1] = step_idx

        if (it_mask == 0).sum() == 0 and n_early_stop &gt; 0:
            print('it %d, all conditions met, stopping' % step_idx)
        return it_mask

    @staticmethod
    def apply_stopvec(it_mask, uncertainty_vec, epistemic_vec, aleatoric_vec, dist_vec, cost_vec, z_vec, x_vec, n_early_stop):
        it_mask = (it_mask - n_early_stop + 1).astype(int)
        for i in range(uncertainty_vec.shape[1]):
            if it_mask[i] &gt; 0 and n_early_stop &gt; 0:
                uncertainty_vec[it_mask[i]:, i] = uncertainty_vec[it_mask[i], i]
                epistemic_vec[it_mask[i]:, i] = epistemic_vec[it_mask[i], i]
                aleatoric_vec[it_mask[i]:, i] = aleatoric_vec[it_mask[i], i]
                cost_vec[it_mask[i]:, i] = cost_vec[it_mask[i], i]
                dist_vec[it_mask[i]:, i] = dist_vec[it_mask[i], i]
                z_vec[it_mask[i]:, i] = z_vec[it_mask[i], i]
                x_vec[it_mask[i]:, i] = x_vec[it_mask[i], i]
        return uncertainty_vec, epistemic_vec, aleatoric_vec, dist_vec, cost_vec, z_vec, x_vec

    def sample_explanations(self, n_explanations, init_std=0.15, min_steps=3, max_steps=25,
                            n_early_stop=3):
        full_x_vec = []
        full_z_vec = []
        full_uncertainty_vec = []
        full_aleatoric_vec = []
        full_epistemic_vec = []
        full_dist_vec = []
        full_cost_vec = []

        for i in range(n_explanations):
            self.randomise_z_init(std=init_std)

            torch.autograd.set_detect_anomaly(False)

            z_vec, x_vec, uncertainty_vec, epistemic_vec, aleatoric_vec, cost_vec, dist_vec = self.optimise(
                min_steps=min_steps, max_steps=max_steps,
                n_early_stop=n_early_stop)

            full_x_vec.append(x_vec)
            full_z_vec.append(z_vec)
            full_uncertainty_vec.append(uncertainty_vec)
            full_aleatoric_vec.append(aleatoric_vec)
            full_epistemic_vec.append(epistemic_vec)
            full_dist_vec.append(dist_vec)
            full_cost_vec.append(cost_vec)

        full_x_vec = np.concatenate(np.expand_dims(full_x_vec, axis=0), axis=0)
        full_z_vec = np.concatenate(np.expand_dims(full_z_vec, axis=0), axis=0)
        full_cost_vec = np.concatenate(np.expand_dims(full_cost_vec, axis=0), axis=0)
        full_dist_vec = np.concatenate(np.expand_dims(full_dist_vec, axis=0), axis=0)
        full_uncertainty_vec = np.concatenate(np.expand_dims(full_uncertainty_vec, axis=0), axis=0)
        full_aleatoric_vec = np.concatenate(np.expand_dims(full_aleatoric_vec, axis=0), axis=0)
        full_epistemic_vec = np.concatenate(np.expand_dims(full_epistemic_vec, axis=0), axis=0)

        return full_x_vec, full_z_vec, full_uncertainty_vec, full_aleatoric_vec, full_epistemic_vec, full_dist_vec, full_cost_vec

    @classmethod
    def batch_optimise(cls, VAE, BNN, original_x, uncertainty_weight, aleatoric_weight, epistemic_weight, prior_weight,
                       distance_weight, latent_L2_weight, prediction_similarity_weight, lr, min_steps=3, max_steps=25,
                       n_early_stop=3, batch_size=256, cond_mask=None, desired_preds=None,
                       distance_metric=None, z_init=None, norm_MNIST=False, flatten_BNN=False, regression=False,
                       prob_BNN=True, cuda=True):
        full_x_vec = []
        full_z_vec = []
        full_uncertainty_vec = []
        full_aleatoric_vec = []
        full_epistemic_vec = []
        full_dist_vec = []
        full_cost_vec = []

        idx_iterator = generate_ind_batch(original_x.shape[0], batch_size=batch_size, random=False, roundup=True)
        for train_idx in idx_iterator:

            if z_init is not None:
                z_init_use = z_init[train_idx]
            else:
                z_init_use = z_init

            if desired_preds is not None:
                desired_preds_use = desired_preds[train_idx].data
            else:
                desired_preds_use = desired_preds

            CLUE_runner = cls(VAE, BNN, original_x[train_idx], uncertainty_weight, aleatoric_weight, epistemic_weight, prior_weight, distance_weight,
                              latent_L2_weight, prediction_similarity_weight, lr, cond_mask=cond_mask, distance_metric=distance_metric,
                              z_init=z_init_use, norm_MNIST=norm_MNIST, desired_preds=desired_preds_use,
                              flatten_BNN=flatten_BNN, regression=regression, prob_BNN=prob_BNN, cuda=cuda)

            z_vec, x_vec, uncertainty_vec, epistemic_vec, aleatoric_vec, cost_vec, dist_vec = \
                CLUE_runner.optimise(min_steps=min_steps, max_steps=max_steps, n_early_stop=n_early_stop)

            full_x_vec.append(x_vec)
            full_z_vec.append(z_vec)
            full_uncertainty_vec.append(uncertainty_vec)
            full_aleatoric_vec.append(aleatoric_vec)
            full_epistemic_vec.append(epistemic_vec)
            full_dist_vec.append(dist_vec)
            full_cost_vec.append(cost_vec)

        full_x_vec = np.concatenate(full_x_vec, axis=1)
        full_z_vec = np.concatenate(full_z_vec, axis=1)
        full_cost_vec = np.concatenate(full_cost_vec, axis=1)
        full_dist_vec = np.concatenate(full_dist_vec, axis=1)
        full_uncertainty_vec = np.concatenate(full_uncertainty_vec, axis=1)
        full_aleatoric_vec = np.concatenate(full_aleatoric_vec, axis=1)
        full_epistemic_vec = np.concatenate(full_epistemic_vec, axis=1)

        return full_x_vec, full_z_vec, full_uncertainty_vec, full_aleatoric_vec, full_epistemic_vec, full_dist_vec, full_cost_vec


class conditional_CLUE(CLUE):

    def __init__(self, VAEAC, BNN, original_x, uncertainty_weight, aleatoric_weight, epistemic_weight, prior_weight, distance_weight,
                 lr, cond_mask=None, distance_metric=None, z_init=None, norm_MNIST=False, flatten_BNN=False,
                 regression=False, cuda=True):

        super(conditional_CLUE, self).__init__(VAEAC, BNN, original_x, uncertainty_weight, aleatoric_weight, epistemic_weight,
                                               prior_weight, distance_weight,
                                               lr, cond_mask, distance_metric, z_init, norm_MNIST, flatten_BNN,
                                               regression, cuda)
        self.cond_mask = cond_mask.type(original_x.type())
        self.VAEAC = VAEAC
        self.prior_weight = 0

    def uncertainty_from_z(self):

        x = self.VAEAC.regenerate(self.z, grad=True)
        x = x * self.cond_mask + self.original_x * (1 - self.cond_mask)

        if self.flatten_BNN:
            to_BNN = x.view(x.shape[0], -1)
        else:
            to_BNN = x

        if self.norm_MNIST:
            to_BNN = calculate_mnist_mean_std_norm(to_BNN)

        if self.regression:
            mu_vec, std_vec = self.BNN.sample_predict(to_BNN, num_samples=0, grad=True)
            total_uncertainty, aleatoric_uncertainty, epistemic_uncertainty = decompose_std_gauss(mu_vec, std_vec)
        else:
            probs = self.BNN.sample_predict(to_BNN, num_samples=0, grad=True)
            total_uncertainty, aleatoric_uncertainty, epistemic_uncertainty = decompose_entropy_cat(probs)

        return total_uncertainty, aleatoric_uncertainty, epistemic_uncertainty, x

    def optimise(self, min_steps=3, max_steps=25,
                 n_early_stop=3):
        # Vectors to capture changes for this minibatch
        z_vec = [self.z.data.cpu().numpy()]
        x_vec = []
        uncertainty_vec = np.zeros((max_steps, self.z.shape[0]))
        aleatoric_vec = np.zeros((max_steps, self.z.shape[0]))
        epistemic_vec = np.zeros((max_steps, self.z.shape[0]))
        dist_vec = np.zeros((max_steps, self.z.shape[0]))
        cost_vec = np.zeros((max_steps, self.z.shape[0]))  # this one doesnt consider the prior

        it_mask = np.zeros(self.z.shape[0])

        for step_idx in range(max_steps):
            self.optimizer.zero_grad()
            total_uncertainty, aleatoric_uncertainty, epistemic_uncertainty, x = self.uncertainty_from_z()
            objective, w_dist = self.get_objective(x, total_uncertainty, aleatoric_uncertainty, epistemic_uncertainty)
            objective.mean(dim=0).backward()  # backpropagate

            self.optimizer.step()

            # save vectors
            uncertainty_vec[step_idx, :] = total_uncertainty.data.cpu().numpy()
            aleatoric_vec[step_idx, :] = aleatoric_uncertainty.data.cpu().numpy()
            epistemic_vec[step_idx, :] = epistemic_uncertainty.data.cpu().numpy()
            dist_vec[step_idx, :] = (w_dist.data.cpu().numpy())
            cost_vec[step_idx, :] = (objective.data.cpu().numpy())
            x_vec.append(x.data)  # we dont convert to numpy yet because we need x0 for L1
            z_vec.append(self.z.data.cpu().numpy())  # this one is after gradient update while x is before

            it_mask = CLUE.update_stopvec(cost_vec, it_mask, step_idx, n_early_stop, min_steps)

        #  Generate final (or resulting s sample)

        x = self.VAE.regenerate(self.z, grad=False).data
        x = x * self.cond_mask + self.original_x * (1 - self.cond_mask)
        x_vec.append(x)
        x_vec = [i.cpu().numpy() for i in x_vec]  # convert x to numpy
        x_vec = np.stack(x_vec)
        z_vec = np.stack(z_vec)

        # Recover correct indexes using mask
        uncertainty_vec, epistemic_vec, aleatoric_vec, dist_vec, cost_vec, z_vec, x_vec = \
            CLUE.apply_stopvec(it_mask, uncertainty_vec, epistemic_vec, aleatoric_vec, dist_vec, cost_vec, z_vec,
                               x_vec, n_early_stop)
        return z_vec, x_vec, uncertainty_vec, epistemic_vec, aleatoric_vec, cost_vec, dist_vec


def decompose_std_gauss(mu, sigma, sum_dims=True):
    aleatoric_var = (sigma ** 2).mean(dim=0)
    epistemic_var = ((mu ** 2).mean(dim=0) - mu.mean(dim=0) ** 2)
    total_var = aleatoric_var + epistemic_var
    if sum_dims:
        aleatoric_var = aleatoric_var.sum(dim=1)
        epistemic_var = epistemic_var.sum(dim=1)
        total_var = total_var.sum(dim=1)
    return total_var.sqrt(), aleatoric_var.sqrt(), epistemic_var.sqrt()


def decompose_entropy_cat(probs, eps=1e-10):
    posterior_preds = probs.mean(dim=0, keepdim=False)
    total_entropy = -(posterior_preds * torch.log(posterior_preds + eps)).sum(dim=1, keepdim=False)

    sample_preds_entropy = -(probs * torch.log(probs + eps)).sum(dim=2, keepdim=False)
    aleatoric_entropy = sample_preds_entropy.mean(dim=0, keepdim=False)

    epistemic_entropy = total_entropy - aleatoric_entropy

    # returns (batch_size)
    return total_entropy, aleatoric_entropy, epistemic_entropy
</code></pre>
<p>gaussian_mlp.py</p>
<pre><code class="lang-auto">class GaussianMLP(nn.Module):
    def __init__(self, input_dim, width, depth, output_dim, flatten_image):
        super(GaussianMLP, self).__init__()

        self.input_dim = input_dim
        self.output_dim = output_dim
        self.width = width
        self.depth = depth
        self.flatten_image = flatten_image

        layers = [nn.Linear(input_dim, width), nn.ReLU()]
        for i in range(depth - 1):
            layers.append(nn.Linear(width, width))
            layers.append(nn.ReLU())
        layers.append(nn.Linear(width, 2 * output_dim))

        self.block = nn.Sequential(*layers)

    def forward(self, x):
        if self.flatten_image:
            x = x.view(-1, self.input_dim)
        x = self.block(x)
        mu = x[:, :self.output_dim]
        sigma = F.softplus(x[:, self.output_dim:])
        return mu, sigma

</code></pre>
<p>gaussian_bnn.py</p>
<pre><code class="lang-auto">class GaussianBNN(BaseNet):
    def __init__(self, model, N_train, lr=1e-2, cuda=True, eps=1e-3, grad_std_mul=20):
        super(GaussianBNN, self).__init__()
        # removed constructor for minimalizm
        self._create_network()
        self._create_optimizer()
        self.schedule = None  # [] #[50,200,400,600]
        self.epoch = 0

        self.grad_buff = []
        self.grad_std_mul = grad_std_mul
        self.max_grad = 1e20
        self.eps = eps

        self.weight_set_samples = []

    def _create_network(self):
        torch.manual_seed(42)
        if self.cuda:
            torch.cuda.manual_seed(42)

        if self.cuda:
            self.model.cuda()
            cudnn.benchmark = True

        print('Total params: %.2fM' % (self.get_nb_parameters() / 1000000.0))

    def _create_optimizer(self):
        self.optimizer = StochasticHamiltonMonteCarloSampler(params=self.model.parameters(), lr=self.lr, base_C=0.05, gauss_sig=0.1)

    def fit(self, x, y, burn_in=False, resample_momentum=False, resample_prior=False):
        self.set_model_mode(train=True)
        x, y = variable_to_tensor_list(variables=(x, y), cuda=self.cuda)

        self.optimizer.zero_grad()
        mu, sigma = self.model(x)
        sigma = sigma.clamp(min=self.eps)
        loss = -diagonal_gauss_loglike(y, mu, sigma).mean(dim=0) * self.N_train

        loss.backward()

        if len(self.grad_buff) &gt; 100:
            self.max_grad = np.mean(self.grad_buff) + self.grad_std_mul * np.std(self.grad_buff)
            self.grad_buff.pop(0)

        self.grad_buff.append(nn.utils.clip_grad_norm_(parameters=self.model.parameters(),
                                                       max_norm=self.max_grad, norm_type=2))
        if self.grad_buff[-1] &gt;= self.max_grad:
            print(self.max_grad, self.grad_buff[-1])
            self.grad_buff.pop()

        self.optimizer.step(burn_in=burn_in, resample_momentum=resample_momentum, resample_prior=resample_prior)

        return loss.data * x.shape[0] / self.N_train, mu.data, sigma.data

    def eval(self, x, y):
        self.set_model_mode(train=False)
        x, y = variable_to_tensor_list(variables=(x, y), cuda=self.cuda)
        mu, sigma = self.model(x)
        sigma = sigma.clamp(min=self.eps)
        loss = -diagonal_gauss_loglike(y, mu, sigma).mean(dim=0) * self.N_train

        return loss.data * x.shape[0] / self.N_train, mu.data, sigma.data

    @staticmethod
    def unnormalised_eval(pred_mu, pred_std, y, y_mu, y_std, gmm=False):
        ll = gaussian_mixture_model_loglike(pred_mu, pred_std, y, y_mu, y_std, gmm=gmm)  # this already computes sum
        if gmm:
            pred_mu = pred_mu.mean(dim=0)
        rms = get_root_mean_square(pred_mu, y, y_mu, y_std)  # this already computes sum
        return rms, ll

    def predict(self, x):
        self.set_model_mode(train=False)
        x, = variable_to_tensor_list(variables=(x,), cuda=self.cuda)
        mu, sigma = self.model(x)
        return mu.data, sigma.data

    def save_sampled_net(self, max_samples):

        if len(self.weight_set_samples) &gt;= max_samples:
            self.weight_set_samples.pop(0)

        self.weight_set_samples.append(copy.deepcopy(self.model.state_dict()))

        logger.warning(f"Saving Samples: {len(self.weight_set_samples)} for Max Samples: {max_samples}")
        logger.warning(f"Samples: {self.weight_set_samples}")

        return None

    def sample_predict(self, x, num_samples, grad=False):
        self.set_model_mode(train=False)
        if num_samples == 0:
            num_samples = len(self.weight_set_samples)

        x, = variable_to_tensor_list(variables=(x,), cuda=self.cuda)

        if grad:
            self.optimizer.zero_grad()
            if not x.requires_grad:
                x.requires_grad = True

        mu_vec = x.data.new(num_samples, x.shape[0], self.model.output_dim)
        std_vec = x.data.new(num_samples, x.shape[0], self.model.output_dim)

        # iterate over all saved weight configuration samples
        for idx, weight_dict in enumerate(self.weight_set_samples):
            if idx == num_samples:
                break
            self.model.load_state_dict(weight_dict)
            # mu_vec[idx], std_vec[idx] = self.model(x)
            mu, std = self.model(x)
            # mu_vec, std_vec = mu_vec.copy(), std_vec.copy()
            mu_vec[idx], std_vec[idx] = mu, std

        if grad:
            return mu_vec[:idx], std_vec[:idx]
        else:
            return mu_vec[:idx].data, std_vec[:idx].data

    def get_weight_samples(self, Nsamples=0):
        weight_vec = []

        if Nsamples == 0 or Nsamples &gt; len(self.weight_set_samples):
            Nsamples = len(self.weight_set_samples)

        for idx, state_dict in enumerate(self.weight_set_samples):
            if idx == Nsamples:
                break

            for key in state_dict.keys():
                if 'weight' in key:
                    weight_mtx = state_dict[key].cpu().data
                    for weight in weight_mtx.view(-1):
                        weight_vec.append(weight)

        return np.array(weight_vec)

    def save_weights(self, filename):
        save_object(self.weight_set_samples, filename)

    def load_weights(self, filename):
        self.weight_set_samples = load_object(filename)
</code></pre>
<p>Full Traceback of the Error Message:</p>
<pre><code class="lang-auto">/Users/lukasscholz/repositorys/counterfactuals/venv/lib/python3.10/site-packages/torch/autograd/__init__.py:266: UserWarning: Error detected in AddmmBackward0. Traceback of forward call that caused the error:
  File "/Users/lukasscholz/Applications/PyCharm Professional Edition.app/Contents/plugins/python/helpers/pydev/pydevd.py", line 2235, in &lt;module&gt;
    main()
  File "/Users/lukasscholz/Applications/PyCharm Professional Edition.app/Contents/plugins/python/helpers/pydev/pydevd.py", line 2217, in main
    globals = debugger.run(setup['file'], None, None, is_module)
  File "/Users/lukasscholz/Applications/PyCharm Professional Edition.app/Contents/plugins/python/helpers/pydev/pydevd.py", line 1527, in run
    return self._exec(is_module, entry_point_fn, module_name, file, globals, locals)
  File "/Users/lukasscholz/Applications/PyCharm Professional Edition.app/Contents/plugins/python/helpers/pydev/pydevd.py", line 1534, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Users/lukasscholz/Applications/PyCharm Professional Edition.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/lukasscholz/repositorys/counterfactuals/examples/test.py", line 210, in &lt;module&gt;
    main()
  File "/Users/lukasscholz/repositorys/counterfactuals/examples/test.py", line 189, in main
    z_vec, x_vec, uncertainty_vec, epistemic_vec, aleatoric_vec, cost_vec, dist_vec = CLUE_explainer.optimise(
  File "/Users/lukasscholz/repositorys/counterfactuals/counterfactual_xai/methods/clue.py", line 191, in optimise
    total_uncertainty, aleatoric_uncertainty, epistemic_uncertainty, x, preds = self.uncertainty_from_z()
  File "/Users/lukasscholz/repositorys/counterfactuals/counterfactual_xai/methods/clue.py", line 124, in uncertainty_from_z
    mu_vec, std_vec = self.BNN.sample_predict(to_BNN, num_samples=0, grad=True)
  File "/Users/lukasscholz/repositorys/counterfactuals/counterfactual_xai/utils/clue/bnn/gaussian_bnn.py", line 134, in sample_predict
    mu, std = self.model(x)
  File "/Users/lukasscholz/repositorys/counterfactuals/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/lukasscholz/repositorys/counterfactuals/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/lukasscholz/repositorys/counterfactuals/counterfactual_xai/utils/clue/gaussian_mlp.py", line 27, in forward
    x = self.block(x)
  File "/Users/lukasscholz/repositorys/counterfactuals/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/lukasscholz/repositorys/counterfactuals/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/lukasscholz/repositorys/counterfactuals/venv/lib/python3.10/site-packages/torch/nn/modules/container.py", line 217, in forward
    input = module(input)
  File "/Users/lukasscholz/repositorys/counterfactuals/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/Users/lukasscholz/repositorys/counterfactuals/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
  File "/Users/lukasscholz/repositorys/counterfactuals/venv/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 116, in forward
    return F.linear(input, self.weight, self.bias)
 (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:118.)
  Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
Traceback (most recent call last):
  File "/Users/lukasscholz/Applications/PyCharm Professional Edition.app/Contents/plugins/python/helpers/pydev/pydevd.py", line 1534, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/Users/lukasscholz/Applications/PyCharm Professional Edition.app/Contents/plugins/python/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/Users/lukasscholz/repositorys/counterfactuals/examples/test.py", line 210, in &lt;module&gt;
    main()
  File "/Users/lukasscholz/repositorys/counterfactuals/examples/test.py", line 189, in main
    z_vec, x_vec, uncertainty_vec, epistemic_vec, aleatoric_vec, cost_vec, dist_vec = CLUE_explainer.optimise(
  File "/Users/lukasscholz/repositorys/counterfactuals/counterfactual_xai/methods/clue.py", line 194, in optimise
    objective.sum(dim=0).backward()  # backpropagate
  File "/Users/lukasscholz/repositorys/counterfactuals/venv/lib/python3.10/site-packages/torch/_tensor.py", line 522, in backward
    torch.autograd.backward(
  File "/Users/lukasscholz/repositorys/counterfactuals/venv/lib/python3.10/site-packages/torch/autograd/__init__.py", line 266, in backward
    Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [200, 2]], which is output 0 of AsStridedBackward0, is at version 1401; expected version 1400 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>

KFrank: <p>Hi Lukas!</p>
<aside class="quote no-group" data-username="Lukas_Scholz" data-post="1" data-topic="201316" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lukas_scholz/48/51053_2.png" class="avatar"> Lukas_Scholz:</div>
<blockquote>
<p>When trying to to a backward-pass in my model I receive the following error:</p>
</blockquote>
</aside>
<p>First, try using the inplace-modification-error debugging suggestions in the<br>
following post:</p>
<aside class="quote quote-modified" data-post="7" data-topic="171826">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation-torch-floattensor-64-1-which-is-output-0-of-asstridedbackward0-is-at-version-3-expected-version-2-instead-hint-the-backtrace-further-a/171826/7">"RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [64, 1]], which is output 0 of AsStridedBackward0, is at version 3; expected version 2 instead. Hint: the backtrace further a</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" style="--category-badge-color: #AB9364; --category-badge-text-color: #FFFFFF;" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Hi Fahmyadan and Sangyoon! 

Here are some suggestions about how to track down (and maybe fix) 
inplace-modification errors.  Note that an inplace modification in the forward 
pass is not necessarily* an error – it depends on whether and how the tensor 
that was modified is used in the backward pass.  Note that inplace operations 
can be useful for saving memory – if you replace an innocent inplace operation 
with an out-of-place equivalent, your training will use more memory (and, to a 
minor e…
  </blockquote>
</aside>

<aside class="quote no-group">
<blockquote>
<p>clue.py</p>
<pre><code class="lang-auto">class CLUE(BaseNet):
    ...
    def optimise(self, min_steps=3, max_steps=25,
                 n_early_stop=3):
            objective.sum(dim=0).backward()

            self.optimizer.step()
            ...
</code></pre>
</blockquote>
</aside>
<p>Note that <code>optimizer.step()</code> counts as an inplace modification for the parameters<br>
being optimized.  Parts of your post have the flavor of <code>optimizer.step()</code> being<br>
the cause of the inplace modification.</p>
<aside class="quote no-group">
<blockquote>
<p>gaussian_mlp.py</p>
<pre><code class="lang-auto">class GaussianMLP(nn.Module):
    def __init__(self, input_dim, width, depth, output_dim, flatten_image):
        ...
        layers = [nn.Linear(input_dim, width), nn.ReLU()]
        for i in range(depth - 1):
            layers.append(nn.Linear(width, width))
            layers.append(nn.ReLU())
        layers.append(nn.Linear(width, 2 * output_dim))
        ...
</code></pre>
<p>Full Traceback of the Error Message:</p>
<pre><code class="lang-auto">...
  File "/Users/lukasscholz/repositorys/counterfactuals/counterfactual_xai/utils/clue/gaussian_mlp.py", line 27, in forward
    x = self.block(x)
  ...
  File "/Users/lukasscholz/repositorys/counterfactuals/venv/lib/python3.10/site-packages/torch/nn/modules/linear.py", line 116, in forward
    return F.linear(input, self.weight, self.bias)
 (Triggered internally at /Users/runner/work/pytorch/pytorch/pytorch/torch/csrc/autograd/python_anomaly_mode.cpp:118.)
  ...
    objective.sum(dim=0).backward()  # backpropagate
  ...
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [200, 2]], which is output 0 of AsStridedBackward0, is at version 1401; expected version 1400 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!
</code></pre>
</blockquote>
</aside>
<p>This error message is telling you that the tensor that was modified inplace has<br>
shape <code>[200, 2]</code>, and it appears to be a <code>Linear</code>.  Does <code>GaussianMLP.block</code><br>
contain a <code>Linear (200, 2)</code>, perhaps the last <code>Linear</code> in the <code>Sequential</code>?</p>
<p>(Note, autograd raises the error on the first inplace modification it detects, so,<br>
if this is what’s going on, earlier <code>Linear</code>s in <code>block</code> probably also have been<br>
modified inplace.)</p>
<p>Anyway, given the error message you posted, start by locating any tensors of<br>
shape <code>[200, 2]</code> you have (call them generically <code>t</code>) and print out <code>t._version</code><br>
at various places in your code to use a divide-and-conquer strategy (binary<br>
search) to locate where <code>t._version</code> changes from <code>1400</code> to <code>1401</code>.  This is<br>
where the relevant inplace modification occurs.</p>
<p>If <code>optimizer.step()</code> is the cause, then ask yourself if you are ever calling<br>
<code>.backward()</code> on a quantity that depends on some “old” data, that is, that<br>
depends (in the sense of a still-live autograd computation graph) on some<br>
data that was computed using some model <code>Parameter</code> (perhaps that <code>Linear</code>)<br>
<em>prior</em> to the most recent <code>optimizer.step()</code> call.</p>
<p>(As an aside, <code>.data</code> has been deprecated for public-facing use.  Using it can<br>
confuse autograd and cause errors.  I didn’t notice any <code>.data</code> calls in your<br>
code that looked like they would be causing trouble, but you should rewrite<br>
your code to eliminate them anyway, if only because the semantics of <code>.data</code><br>
could silently change in future versions.)</p>
<p>Good luck!</p>
<p>K. Frank</p>

Lukas_Scholz: <p>Thanks for the reply <a class="mention" href="/u/kfrank">@KFrank</a> !<br>
I searched the last two days for the [200, 2] Tensor you mentioned, but had no luck in finding it.<br>
Since the original <a href="https://github.com/cambridge-mlg/CLUE" rel="noopener nofollow ugc">CLUE</a> project is implemented in PyTorch 1.3.1 and Python 2.7 I tried to get it running with those dependencies and everything worked fine with the same code. My code base is build upon python&gt;=3.8 and torch 2.1.0</p>
<p>Do you maybe have an intuition why the inplace change of the tensor is happening with newer pytorch and python versions?</p>

KFrank: <p>Hi Lukas!</p>
<aside class="quote no-group" data-username="Lukas_Scholz" data-post="3" data-topic="201316" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lukas_scholz/48/51053_2.png" class="avatar"> Lukas_Scholz:</div>
<blockquote>
<p>I searched the last two days for the [200, 2] Tensor you mentioned, but had no luck in finding it.</p>
</blockquote>
</aside>
<p>Keep looking for it.  Finding the tensor that is being modified inplace opens up<br>
probably the best avenue for debugging the issue.  (I’ve never seen autograd<br>
“hallucinate” the shape of a tensor it flags as having been modified inplace.<br>
So I really bet you have such a tensor in your model.)</p>
<p>Note that a <code>Linear (200, 2)</code> has a weight of shape <code>[2, 200]</code> and autograd<br>
will typically report its <em>transpose</em> (with shape <code>[200, 2]</code>) as being modified<br>
inplace.  So if you have a <code>Linear.weight</code> with shape <code>[2, 200]</code>, that would<br>
likely be your culprit.</p>
<p>I would recommend that you print out the shapes of the inputs, outputs and<br>
<code>weight</code>s of the <code>Linear</code>s that make up <code>GaussianMLP.block</code>.  The forward-call<br>
traceback you posted strongly suggests that one of the tensors in <code>block</code> is<br>
the one that gets modified inplace.</p>
<p>Note, the “version 1401” in your error message is potentially meaningful.  This<br>
suggests to me that the tensor in question is one being optimized, that you’ve<br>
called <code>optimizer.step()</code> 1399 times (modifying the tensor inpace) without<br>
error, but after your 1400th call, the error, for whatever reason, shows up.  (For<br>
example, do you have 1400 batches in an epoch, but then do something a bit<br>
different at the end of an epoch (or at the beginning of the next epoch?)</p>
<p>Also, if you call into any third-party code, you might need to “instrument” that<br>
code as well (that is, add code to it that prints out the shapes of various tensors).</p>
<aside class="quote no-group">
<blockquote>
<p>Do you maybe have an intuition why the inplace change of the tensor is happening with newer pytorch and python versions?</p>
</blockquote>
</aside>
<p>I don’t see any obvious explanation for this.   1.3.1 is pretty old by now.  It’s<br>
possible that a bug in the old version somehow let the inplace-modification<br>
error slip by, but the newer version catches it.</p>
<p>Also (although, again, I don’t see any issues with your use of <code>.data</code>), a<br>
backward-breaking change in the semantics of <code>.data</code> could hypothetically<br>
make the two versions behave differently.  (In any event, if only as a matter<br>
of good housekeeping, you should get rid of the use of <code>.data</code> in your code.)</p>
<p>Best.</p>
<p>K. Frank</p>

Lukas_Scholz: <p><a class="mention" href="/u/kfrank">@KFrank</a> I found the tensor and could trace back it’s origin, but had no luck with using <code>clone()</code> or removing inplace operators like <code>add_()</code>. Here is the way the tensors are created:</p>
<ol>
<li><code>total_uncertainty, aleatoric_uncertainty, epistemic_uncertainty, x, preds = self.uncertainty_from_z()</code> creats the tensor <code>x</code></li>
<li>Inside <code>uncertainty_from_z()</code> the tensor <code>x</code> is created like this <code>x = self.VAE.regenerate(self.z, grad=True)</code>, there <code>z</code> is a tensor of shape <code>[512, 16]</code> which holds data from my valset, that is created earlier, output shape is <code>[512, 12]</code></li>
<li>Then tensor <code>x</code> is fed into my GaussianBNN <code>mu_vec, std_vec = self.BNN.sample_predict(x, num_samples=0, grad=True)</code> the function <code>sample_predict()</code> looks like this:</li>
</ol>
<pre><code class="lang-auto">    def sample_predict(self, x, num_samples, grad=False):
        self.set_model_mode(train=False)
        if num_samples == 0:
            num_samples = len(self.weight_set_samples)
        x, = variable_to_tensor_list(variables=(x,), cuda=self.cuda) # torch.Size([512, 12])
        if grad:
            self.optimizer.zero_grad()
            if not x.requires_grad:
                x.requires_grad = True
        mu_vec = x.new(num_samples, x.shape[0], self.model.output_dim) # torch.Size([100, 512, 1])
        std_vec = x.new(num_samples, x.shape[0], self.model.output_dim) # torch.Size([100, 512, 1])
        for idx, weight_dict in enumerate(self.weight_set_samples): # 100 iterations
            if idx == num_samples:
                break
            self.model.load_state_dict(weight_dict)
            mu, std = self.model(x.clone()) # x = torch.Size([512, 12])
            mu_vec[idx] = mu.clone() # torch.Size([99, 512, 1])
            std_vec[idx] = std.clone() # torch.Size([99, 512, 1])

        if grad:
            return mu_vec[:idx].clone(), std_vec[:idx].clone() # both torch.Size([99, 512, 1])
        else:
            return mu_vec[:idx].clone(), std_vec[:idx].clone()
</code></pre>
<p>Note: The for loop in <code>sample_predict()</code> is being called before that 13-times, the 14th time is from the <code>uncertainty_from_z()</code>-function for 100 iterations. All other 13 calls also iterated 100 times. So the 14th iteration would be the create/modify the version <code>1400</code> of the tensor from the error traceback.</p>
<ol start="4">
<li><code>self.model(x.clone())</code> calls the GaussianMLP you mentioned. There the forward pass is called. This is executed 99 times instead of 100, therefore the <code>mu_vec</code> and <code>std_vec</code> are <code>torch.Size([99, 512, 1])</code> instead of <code>torch.Size([100, 512, 1])</code>.<br>
There the forward pass looks like this:</li>
</ol>
<pre><code class="lang-auto">    def forward(self, x):
        if self.flatten_image:
            x = x.view(-1, self.input_dim)


        for i, layer in enumerate(self.block):
            x = layer(x) # input is torch.Size([512, 12])
            # after last layer output size is torch.Size([512, 2]
            # ...and weight.shape is torch.Size([2, 200]) OUR TENSOR IN SEARCH 
            if isinstance(layer, nn.Linear):
                print("Weight shape for layer {}: {}".format(i, layer.weight.shape))

        mu = x[:, :self.output_dim]
        sigma = F.softplus(x[:, self.output_dim:])
        return mu, sigma
</code></pre>
<ol start="4">
<li>After going through the forward pass and the whole GaussianBNN the function <code>uncertainty_from_z()</code> takes the <code>mu_vec</code>and <code>std_vec</code>, which should have one more dimension and computes an uncertainty metric based on it:</li>
</ol>
<pre><code class="lang-auto">                mu_vec, std_vec = self.BNN.sample_predict(to_BNN, num_samples=0, grad=True)
                total_uncertainty, aleatoric_uncertainty, epistemic_uncertainty = decompose_std_gauss(mu_vec, std_vec)
                preds = mu_vec.mean(dim=0)
                print(f"PREDS: {preds.shape}")
            
        else:
            if self.regression:
                mu, std = self.BNN.predict(to_BNN, grad=True)
                total_uncertainty = std.squeeze(1)
                aleatoric_uncertainty = total_uncertainty
                epistemic_uncertainty = total_uncertainty * 0
                preds = mu

        return total_uncertainty, aleatoric_uncertainty, epistemic_uncertainty, x, preds
</code></pre>
<ol start="5">
<li>
<p>The final step before the backward pass <code>objective.sum(dim=0).backward()</code> is getting the objective with <code>objective, w_dist = self.get_objective(x, total_uncertainty, aleatoric_uncertainty, epistemic_uncertainty, preds)</code> The function <code>get_objective()</code>. The <code>objective</code> and <code>w_dist</code> both have the shape <code>torch.Size([512])</code></p>
</li>
<li>
<p>After that <code>objective.sum(dim=0).backward()</code> happens and I get the error.</p>
</li>
</ol>
<p>Sorry for all the text, but I’m really clueless how I can debug this problem, since I tried to <code>.clone()</code> basically every tensor and eliminate all inplace operations like <code>add_()</code> or inplace modifications like:</p>
<pre><code class="lang-auto">mu_vec[idx], std_vec[idx] = self.model(x)
</code></pre>
<p>Is the missing dim in <code>mu_vec</code> and <code>std_vec</code> the root case of the issue? How can I further debug this?</p>

KFrank: <p>Hi Lukas!</p>
<p>You have a lot of things going on here …</p>
<p>First, to confirm, does your error always show up with “expected version 1400,”<br>
or does the error’s “expected version” vary somewhat?</p>
<aside class="quote no-group" data-username="Lukas_Scholz" data-post="5" data-topic="201316" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/lukas_scholz/48/51053_2.png" class="avatar"> Lukas_Scholz:</div>
<blockquote>
<pre><code class="lang-auto">    def sample_predict(self, x, num_samples, grad=False):
        self.set_model_mode(train=False)
</code></pre>
</blockquote>
</aside>
<p>Just to be clear, your <code>set_model_mode (train = False)</code> is probably only turning<br>
off things like <code>Dropout</code> and not turning off autograd tracking.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">        for idx, weight_dict in enumerate(self.weight_set_samples): # 100 iterations
            if idx == num_samples:
                break
            self.model.load_state_dict(weight_dict)
</code></pre>
</blockquote>
</aside>
<p>Note that <code>model.load_state_dict()</code> modifies the <code>model</code>’s <code>parameters</code> inplace.<br>
So you should track this as the possible cause of your error.</p>
<aside class="quote no-group">
<blockquote>
<p>Note: The for loop in <code>sample_predict()</code> is being called before that 13-times, the 14th time is from the <code>uncertainty_from_z()</code>-function for 100 iterations. All other 13 calls also iterated 100 times. So the 14th iteration would be the create/modify the version <code>1400</code> of the tensor from the error traceback.</p>
</blockquote>
</aside>
<p>Just based on this numerology, this seems likely to be related to you error.</p>
<p>What is being done on the 1401st “iteration” that is different than the first 1400?</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">    def forward(self, x):
        ...
        for i, layer in enumerate(self.block):
            x = layer(x) # input is torch.Size([512, 12])
            # after last layer output size is torch.Size([512, 2]
            # ...and weight.shape is torch.Size([2, 200]) OUR TENSOR IN SEARCH 
</code></pre>
</blockquote>
</aside>
<p>This looks correct.  Now you need to determining where this tensor is being<br>
modified for the 1401st time.</p>
<p>From outside <code>GaussianMLP.forward()</code> this tensor can presumably be accessed<br>
as something like <code>model.block[-1].weight</code>.  (Note, the other <code>Linear</code>’s in<br>
<code>block</code> probably also have the same issue.  It’s just that autograd stops looking<br>
after the first error is detected.)</p>
<p>So place <code>print (model.block[-1].weight._version)</code> at various places<br>
in your code, especially just before and after your calls to</p>
<pre data-code-wrap="python"><code class="lang-python">self.model.load_state_dict(weight_dict)
</code></pre>
<p>and</p>
<pre data-code-wrap="python"><code class="lang-python">self.optimizer.step()
</code></pre>
<p>It’s not that the call that modifies <code>block[-1].weight</code> inplace is necessarily in<br>
error.  Both loading the state dict and calling the optimizer are logically sensible<br>
things to do.</p>
<p>Let me call <em>old data</em> data that was computed using the <code>model</code> prior to the<br>
most recent modification to <code>block[-1].weight</code>.  You are almost certainly<br>
calling <code>.backward()</code> on some quantity that depends on some old data.</p>
<p>If this is the case, why is it happening, and why doesn’t it happen until the<br>
1401st iteration?  What difference in the code path on the 1401st iteration<br>
is causing that old data to slip in?</p>
<p>If this is the cause of your error, in can be “fixed” by <code>.detach()</code>ing the old<br>
data before using it in backward.  Something like:</p>
<pre data-code-wrap="python"><code class="lang-python">objective = some_new_objective + old_data.detach()
objective.sum(dim=0).backward()
</code></pre>
<p>Because of the <code>.detach()</code>, this code will not backpropagate through <code>old_data</code><br>
and the inplace-modification error won’t occur.</p>
<p>But this might not be <em>logically</em> correct.  It’s up to the details of your use case<br>
whether the gradients computed by <code>objective.sum (dim = 0).backward()</code><br>
should include the gradients due to <code>old_data</code>.</p>
<aside class="quote no-group">
<blockquote>
<p>eliminate all inplace operations like <code>add_()</code> or inplace modifications like:</p>
<pre><code class="lang-auto">mu_vec[idx], std_vec[idx] = self.model(x)
</code></pre>
</blockquote>
</aside>
<p>As you note, <code>mu_vec[idx], std_vec[idx] = self.model(x)</code> does modify<br>
<code>mu_vec</code> and <code>std_vec</code> inplace, but this would seem not to be the (immediate)<br>
cause of your issue because it isn’t modifying <code>model.block[-1].weight</code><br>
inplace.</p>
<p>To summarize:</p>
<p>Determine exactly which line of code (in which iterations of any enclosing loops)<br>
is causing <code>_version</code> to be incremented to <code>1401</code>.  The next call to <code>.backward()</code><br>
(on outputs of the relevant model) should be the one that raises the error.</p>
<p>Think hard about what’s different about that 1401st iteration.</p>
<p>Look hard at <code>objective.sum (dim = 0)</code>, the quantity on which you are calling<br>
<code>.backward()</code>.  Is there any way some old data can be creeping into <code>objective</code>?<br>
If there is, think logically about why the old data is there.  Can it be left out?  Can<br>
it be <code>.detached()</code>?</p>
<p>If you logically need gradients from the old data, you may need to modify the<br>
forward pass of <code>model</code> so that autograd can hold on to a copy of the tensor<br>
being modified inplace.  (This can certainly be done, but you would need to<br>
perform some surgery on your model.)</p>
<p>Good luck!</p>
<p>K. Frank</p>