Update multiple models in each batch: modified by an inplace operation Error

soulless032: <p>I try to train multiple models within each batch, since the loss functions are based on the result of all models. Each model has their own dataset and optimizer. However, when I try to do backward() on the second model, “modified by an inplace operation” error occurs. I do set detect_anomaly to True, but the returned traceback makes no sense to me. Here is the code:</p>
<pre><code class="lang-auto">    for epoch in range(num_epochs):
        for i in range(len(splitted_v1)):
            X1_ = model_v1(splitted_v1[i].float())
            S1 = similarity_matrix(X1_)

            X2_ = model_v2(splitted_v2[i].float())
            S2 = similarity_matrix(X2_)

            local_s = (S1 + S2) / 2

            loss_v1 = custom_loss(X1_, splitted_v1[i], S1, local_s)
            optimizer_v1.zero_grad()
            loss_v1.backward(retain_graph=True)
            optimizer_v1.step()

            loss_v2 = custom_loss(X2_, splitted_v2[i], S2, local_s)
            optimizer_v2.zero_grad()
            loss_v2.backward(retain_graph=True)
            optimizer_v2.step()
</code></pre>
<p>Here is the traceback:</p>
<blockquote>
<p>Blockquote<br>
Warning: Error detected in AddmmBackward. Traceback of forward call that caused the error:<br>
File “/Users/xx/Desktop/cancer_research/synthetic_test.py”, line 336, in <br>
X1_ = model_v1(splitted_v1[i].float())<br>
File “/Users/xx/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/Users/xx/Desktop/cancer_research/synthetic_test.py”, line 102, in forward<br>
x = self.decoder(x)<br>
File “/Users/xx/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/Users/xx/anaconda3/lib/python3.7/site-packages/torch/nn/modules/container.py”, line 100, in forward<br>
input = module(input)<br>
File “/Users/xx/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 550, in <strong>call</strong><br>
result = self.forward(*input, **kwargs)<br>
File “/Users/xx/anaconda3/lib/python3.7/site-packages/torch/nn/modules/linear.py”, line 87, in forward<br>
return F.linear(input, self.weight, self.bias)<br>
File “/Users/xx/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py”, line 1610, in linear<br>
ret = torch.addmm(bias, input, weight.t())<br>
(print_stack at /Users/distiller/project/conda/conda-bld/pytorch_1587428061935/work/torch/csrc/autograd/python_anomaly_mode.cpp:60)<br>
Traceback (most recent call last):<br>
File “/Users/xx/Desktop/cancer_research/synthetic_test.py”, line 351, in <br>
loss_v2.backward(retain_graph=True)<br>
File “/Users/xx/anaconda3/lib/python3.7/site-packages/torch/tensor.py”, line 198, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/Users/xx/anaconda3/lib/python3.7/site-packages/torch/autograd/<strong>init</strong>.py”, line 100, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
Blockquote</p>
</blockquote>
<p>How the call on model_v1 influences the variable of model_v2.</p>
<p>I am new to pytorch. Any help would be appreciated!</p>

ptrblck: <p>Are the model updates working separately and is the issue thus raise only in the current setup?<br>
Also, is <code>similarity_matrix</code> or a model manipulating some tensors inplace?</p>

soulless032: <p>Thank you for the replay!<br>
Yes. They work fine separately during the initialization step, and I don’t think I use in-place operation in the similarity_matrix() and custom_loss().<br>
I initialize the models on their own like this:</p>
<pre><code class="lang-auto">    print("initializing model_v1")
    for epoch in range(5):

        for i in range(len(splitted_v1)):
            train = splitted_v1[i]
            # ===================forward=====================
            output = model_v1(train.float())
            loss = torch.norm(output-train, p='fro')
            # ===================backward====================
            optimizer_v1.zero_grad()
            loss.backward()
            optimizer_v1.step()
        # ===================log========================
        print('epoch [{}/{}], loss:{:.4f}'
              .format(epoch + 1, 5, loss))
        lr_scheduler_v1.step()

    print("initializing model_v2")
    for epoch in range(5):

        for i in range(len(splitted_v2)):
            train = splitted_v2[i]
            # ===================forward=====================
            output = model_v2(train.float())
            loss = torch.norm(output - train, p='fro')
            # ===================backward====================
            optimizer_v2.zero_grad()
            loss.backward()
            optimizer_v2.step()
        # ===================log========================
        print('epoch [{}/{}], loss:{:.4f}'
              .format(epoch + 1, 5, loss))
        lr_scheduler_v2.step()
</code></pre>
<p>Here is similarity_matrix:</p>
<pre><code class="lang-auto">def similarity_matrix(y_pred):
    # S = [\hatX_k \dot \hatX_k^T / p_k]
    s = torch.mm(y_pred, y_pred.t())/y_pred.size()[1]
    return s
</code></pre>
<p>Here is custom_loss:</p>
<pre><code class="lang-auto">def custom_loss(y_pred, y_true, s_v, local_s):
    S = similarity_matrix(y_pred)
    loss = torch.tensor(0, requires_grad=True, dtype=float)
    for i in range(S.size(0)):
        x = S[i] + sys.float_info.epsilon
        temp = x * torch.log(x) + (1 - x) * torch.log(1 - x)
        loss = loss - temp.sum()

    loss3 = loss/y_pred.size(0)
    # frobenius norm || \hatX_k - X_k ||^2_F
    loss1 = torch.norm(y_pred-y_true, p='fro')

    loss2 = torch.norm(s_v-local_s, p='fro')
    total_loss = loss1 + loss2 + loss3
    return total_loss
</code></pre>

ptrblck: <p>Thanks for the code. Sorry, but I cannot find the error.<br>
Could you post a minimal, executable code snippet using random input tensors, so that we could debug it, please?</p>

soulless032: <p>I solve this problem. There was nothing wrong in the code, and pytorch and torchvision were up to date. I thought it may be caused by incompatibilities between packages, so I reinstall anaconda and its environment. Then it works. Thank you very much!</p>