RuntimeError: cuda runtime error (710) : device-side assert triggered at

username: Sangwon_Jake
cooked: <blockquote>
<p>Traning the image classification with pytorch <br> got following<br>
error messageK<br></p>
</blockquote>
<hr>
<blockquote>
<p>RuntimeError                              Traceback (most recent call<br>
last)  in <br>
29         print(len(train_loader.dataset),len(valid_loader.dataset))<br>
30         <span class="hashtag-raw">#break</span><br>
—&gt; 31         train_loss, train_acc ,model= train(model, device, train_loader, optimizer, criterion)<br>
32         valid_loss, valid_acc,model = evaluate(model, device, valid_loader, criterion)<br>
33</p>
<p> in train(model, device, iterator,<br>
optimizer, criterion)<br>
21         acc = calculate_accuracy(fx, y)<br>
22         <span class="hashtag-raw">#print</span>(“5.”)<br>
—&gt; 23         loss.backward()<br>
24<br>
25         optimizer.step()</p>
<p>~/venv/lib/python3.7/site-packages/torch/tensor.py in backward(self,<br>
gradient, retain_graph, create_graph)<br>
164                 products. Defaults to <code>False</code>.<br>
165         “”"<br>
 → 166         torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
167<br>
168     def register_hook(self, hook):</p>
<p>~/venv/lib/python3.7/site-packages/torch/autograd/<strong>init</strong>.py in<br>
backward(tensors, grad_tensors, retain_graph, create_graph,<br>
grad_variables)<br>
97     Variable._execution_engine.run_backward(<br>
98         tensors, grad_tensors, retain_graph, create_graph,<br>
—&gt; 99         allow_unreachable=True)  # allow_unreachable flag<br>
100<br>
101</p>
<p>RuntimeError: cuda runtime error (710) : device-side assert triggered<br>
at /pytorch/aten/src/THC/generic/THCTensorMath.cu:26</p>
</blockquote>
<p>Related code block is here <br></p>
<pre><code>def train(model, device, iterator, optimizer, criterion):

print('train')
epoch_loss = 0
epoch_acc = 0

model.train()


for (x, y) in iterator:
    #print(x,y)
    x,y = x.cuda(), y.cuda()
    #x = x.to(device)
    #y = y.to(device)
    #print('1')
    optimizer.zero_grad()
    #print('2')
    fx = model(x)
    #print('3')
    loss = criterion(fx, y)
    #print("4.loss-&gt;",loss)
    acc = calculate_accuracy(fx, y)
    #print("5.")
    loss.backward()
    
    optimizer.step()
    
    epoch_loss += loss.item()
    epoch_acc += acc.item()
    
return epoch_loss / len(iterator), epoch_acc / len(iterator),model


    EPOCHS = 5
    SAVE_DIR = 'models'
    MODEL_SAVE_PATH = os.path.join(SAVE_DIR, 'please.pt')
    from torch.utils.data import DataLoader
    best_valid_loss = float('inf')
    
    if not os.path.isdir(f'{SAVE_DIR}'):
        os.makedirs(f'{SAVE_DIR}')
    print("start")
    for epoch in range(EPOCHS):
        print('================================',epoch ,'================================')
        for i , (train_idx, valid_idx) in enumerate(zip(train_indexes, valid_indexes)):
            print(i,train_idx,valid_idx,len(train_idx),len(valid_idx))
            
            traindf = df_train.iloc[train_index, :].reset_index()
            validdf = df_train.iloc[valid_index, :].reset_index()
            
            #traindf = df_train
            #validdf = df_train
            
            train_dataset = TrainDataset(traindf, mode='train', transforms=data_transforms)
            valid_dataset = TrainDataset(validdf, mode='valid', transforms=data_transforms)
            
            train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)
            valid_loader = DataLoader(valid_dataset, batch_size=batch_size, shuffle=False)
    
    
    
            print(len(train_loader.dataset),len(valid_loader.dataset))
            #break
            train_loss, train_acc ,model= train(model, device, train_loader, optimizer, criterion)
            valid_loss, valid_acc,model = evaluate(model, device, valid_loader, criterion)
            
            if valid_loss &lt; best_valid_loss:
                best_valid_loss = valid_loss
                torch.save(model,MODEL_SAVE_PATH)
    
            print(f'| Epoch: {epoch+1:02} | Train Loss: {train_loss:.3f} | Train Acc: {train_acc*100:05.2f}% | Val. Loss: {valid_loss:.3f} | Val. Acc: {valid_acc*100:05.2f}% |')
</code></pre>
<blockquote>
<p>splits = zip(train_indexes, valid_indexes)<br>
[ 3692  3696  3703 … 30733 30734 30735] [   0    1    2 … 4028<br>
4041 4046] [    0     1     2 … 30733 30734 30735] [3692 3696 3703<br>
… 7986 7991 8005] [    0     1     2 … 30733 30734 30735] [ 7499<br>
7500  7502 … 11856 11858 11860] [    0     1     2 … 30733 30734<br>
30735] [11239 11274 11280 … 15711 15716 15720] [    0     1     2<br>
… 30733 30734 30735] [15045 15051 15053 … 19448 19460 19474] [<br>
0     1     2 … 30733 30734 30735] [18919 18920 18926 … 23392<br>
23400 23402] [    0     1     2 … 30733 30734 30735] [22831 22835<br>
22846 … 27118 27120 27124] [    0     1     2 … 27118 27120 27124]<br>
[26718 26721 26728 … 30733 30734 30735]</p>
</blockquote>

username: ptrblck
cooked: <p>Could you rerun the code with <code>CUDA_LAUNCH_BLOCKING=1 python script.py args</code> and post the stack trace here, please?</p>