Inplace operation Error for simple addition and Subtraction operations

Waqas_Sheikh: <p>When I run the following code with one layer network it works well. However, When I switch to two layer network it generates the Following Error. Can Some help me resolve the issue. The code is cloned from the following GitHub Repo.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/amina01/ESMIL">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/amina01/ESMIL" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/3af2046449f644d3ad11c435c29121b4257bd5af43fe77a8a7a526c2a285401b/amina01/ESMIL" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/amina01/ESMIL" target="_blank" rel="noopener nofollow ugc">GitHub - amina01/ESMIL: An embarrassingly simple approach to neural multiple...</a></h3>

  <p>An embarrassingly simple approach to neural multiple instance classification - GitHub - amina01/ESMIL: An embarrassingly simple approach to neural multiple instance classification</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Thanks</p>
<blockquote>
<p>RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [230, 1]], which is output 0 of TBackward, is at version 2; expected version 1 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).</p>
</blockquote>
<blockquote>
<h1>-<em>- coding: utf-8 -</em>-</h1>
<p>“”"<br>
Created on Fri Feb 22 16:24:22 2019<br>
<span class="mention">@author</span>: Amina<br>
The following code is for one ten fold cross validation run of neural MIL on Tiger dataset.<br>
By default it runs for Single layer architecture.<br>
“”"</p>
<p>import torch.nn as nn<br>
import torch.nn.functional as F<br>
import torch.optim as optim<br>
from torch.autograd import Variable<br>
import torch</p>
<p>import numpy as np<br>
from torch.utils.data import Dataset, DataLoader<br>
from sklearn.metrics import roc_auc_score as auc_roc<br>
from sklearn import metrics<br>
import <a href="http://scipy.io" rel="noopener nofollow ugc">scipy.io</a><br>
from sklearn.model_selection import StratifiedKFold</p>
<p>class MyDataset(Dataset):<br>
def <strong>init</strong>(self, bags):<br>
self.bags = bags</p>
<pre><code>def __getitem__(self, index):
    examples = self.bags[index]
    return examples
    
def __len__(self):
    return len(self.bags)
</code></pre>
<p>‘’’<br>
Single Layer Architecture<br>
‘’’</p>
<p>class Net(nn.Module):<br>
def <strong>init</strong>(self,d):<br>
super(Net, self).<strong>init</strong>()<br>
self.out = nn.Linear(d,1)</p>
<pre><code>def forward(self,x):
    x = x.view(x.size(0), -1)
    x = self.out(x)
    return x
</code></pre>
<p>‘’’<br>
One Hidden Layer Architecture<br>
‘’’</p>
<h1></h1>
<p><span class="hashtag-raw">#class</span> Net(nn.Module):</p>
<h1>def <strong>init</strong>(self,d):</h1>
<h1>super(Net, self).<strong>init</strong>()</h1>
<h1>self.hidden1 = nn.Linear(d,d)</h1>
<h1>self.out = nn.Linear(d,1)</h1>
<h1></h1>
<h1>def forward(self,x):</h1>
<h1>x = x.view(x.size(0), -1)</h1>
<h1>x = self.hidden1(x)</h1>
<h1>x = F.tanh(x)</h1>
<h1></h1>
<h1>x = self.out(x)</h1>
<h1>return x</h1>
<p>def create_bags_mat(path=‘data\elephant_100x100_matlab.mat’):<br>
mat=scipy.io.loadmat(path)<br>
ids=mat[‘bag_ids’][0]<br>
f=scipy.sparse.csr_matrix.todense(mat[‘features’])<br>
l=np.array(scipy.sparse.csr_matrix.todense(mat[‘labels’]))[0]<br>
bags=<span class="chcklst-box fa fa-square-o fa-fw"></span><br>
labels=<span class="chcklst-box fa fa-square-o fa-fw"></span><br>
for i in set(ids):<br>
bags.append(np.array(f[ids==i]))<br>
labels.append(l[ids==i][0])<br>
bags=np.array(bags)<br>
labels=np.array(labels)<br>
return bags, labels</p>
<p>aucs=<span class="chcklst-box fa fa-square-o fa-fw"></span><br>
accs=<span class="chcklst-box fa fa-square-o fa-fw"></span><br>
bags, labels=create_bags_mat()</p>
<p>skf = StratifiedKFold(n_splits=10)<br>
for train, test in skf.split(bags, labels):</p>
<pre><code>bags_tr=bags[train]
y_tr=labels[train]
bags_ts=bags[test]
y_ts=labels[test]
pos_bags=bags_tr[y_tr&gt;0]
neg_bags=bags_tr[y_tr&lt;0]



pos=MyDataset(pos_bags)
neg=MyDataset(neg_bags)

loader_pos = DataLoader(pos, batch_size=1)
loader_neg = DataLoader(neg, batch_size=1)
epochs=10
mlp=Net(230)
mlp.cuda()
</code></pre>
<h1>torch.set_default_tensor_type(‘torch.cuda.FloatTensor’)</h1>
<pre><code>optimizer = optim.Adam(mlp.parameters())

all_losses=[]
for e in range(epochs):
    l=0.0
    for idx_p, pbag in enumerate(loader_pos):
        pbag=pbag.float()
        pbag=Variable(pbag).type(torch.cuda.FloatTensor)
        p_scores=mlp.forward(pbag[0])
        max_p=torch.max(p_scores)

        for idx_n, nbag in enumerate(loader_neg):
            nbag=nbag.float()
            nbag=Variable(nbag).type(torch.cuda.FloatTensor)
            n_scores=mlp.forward(nbag[0])

            max_n=torch.max(n_scores)
            z=np.array([0.0])
            loss=torch.max(Variable(torch.from_numpy(z)).type(torch.cuda.FloatTensor), (max_n-max_p+1))
</code></pre>
<h1>loss=torch.max(torch.tensor(0.0), (max_n-max_p+1))</h1>
<pre><code>            l=l+float(loss)

            optimizer.zero_grad()
            loss.backward(retain_graph=True)

            optimizer.step()
    all_losses.append(l)
#testing

test=MyDataset(bags_ts)
loader_ts=DataLoader(test, batch_size=1)
predictions=[]

for param in mlp.parameters():
    param.requires_grad =False
for idx_ts, tsbag in enumerate(loader_ts):
    tsbag=tsbag.float()
    tsbag=Variable(tsbag).type(torch.cuda.FloatTensor)
    scores=mlp.forward(tsbag[0])

    predictions.append(float(torch.max(scores)))
auc=auc_roc(y_ts, predictions)
aucs.append(auc)
print ('AUC=',auc)


f, t, a=metrics.roc_curve(y_ts, predictions)
AN=sum(x&lt;0 for x in y_ts)
AP=sum(x&gt;0 for x in y_ts)
TN=(1.0-f)*AN
TP=t*AP
Acc2=(TP+TN)/len(y_ts)
acc=max(Acc2)
print ('accuracy=',acc )
accs.append(acc)
</code></pre>
<p>print (“\n\nmean auc=”, np.mean(aucs))<br>
print (“mean acc=”, np.mean(accs))</p>
</blockquote>

pks: <p>i am facing similar issues as well, tried clone() and other solutions suggested, but nothing seems to work.</p>

Zheng_An_Zhu: <p>I add a copy model to deal with <code>p_scores</code> forward process and it works to me in your two layers network.<br>
I have no idea about why original code works in single fc layer, sorry.</p>
<p>New version (related by <a href="https://discuss.pytorch.org/t/solved-pytorch1-5-runtimeerror-one-of-the-variables-needed-for-gradient-computation-has-been-modified-by-an-inplace-operation/90256/22">here</a>):</p>
<pre><code class="lang-auto">mlp_new = deepcopy(mlp)
p_scores=mlp_new.forward(pbag[0])
</code></pre>