Getting "/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed." error while training

Sourabh_Patil: <p>I am training a model where I am using indexing to pull out feature maps. The dimension of feature maps that I get from the backbone is variable, but every time I pool the same number of feature maps. I am getting following error:</p>
<p>dexSelectSmallIndex: block: [99,0,0], thread: [64,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [65,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [66,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [67,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [68,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [69,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [70,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [71,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [72,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [73,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [74,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [75,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [76,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [77,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [78,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [79,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [80,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [81,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [82,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [83,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [84,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [85,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [86,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [87,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [88,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [89,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [90,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [91,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [92,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [93,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [94,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [95,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [96,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [97,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [98,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [99,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [100,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [101,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [102,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [103,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [104,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [105,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [106,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [107,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [108,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [109,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [110,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [111,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [112,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [113,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [114,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [115,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [116,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [117,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [118,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [119,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [120,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [121,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [122,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [123,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [124,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [125,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [126,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [127,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
R18_look_at_particular_patch: 1/1000:   0%|                                                    | 0/1844 [00:11&lt;?, ?it/s]<br>
Traceback (most recent call last):<br>
File “main_2.py”, line 174, in <br>
epoch_losses.update(train_loss.item(), len(x))<br>
RuntimeError: CUDA error: device-side assert triggered<br>
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.<br>
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.</p>
<p>How to resolve it?</p>

ptrblck: <p>The error is raised by a failing indexing operation. Rerun your code via <code>CUDA_LAUNCH_BLOCKING=1</code> to isolate the failing line of code and fix the values in the index tensor.</p>

Sourabh_Patil: <p>Hi! Thanks for the reply. I tried that but I am getting a similar error. It seems that it is related to indexing.</p>
<p>Error tail part:</p>
<p>/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [49,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [50,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [51,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [52,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [53,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [54,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [55,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [56,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [57,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [58,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [59,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [60,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [61,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [62,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [63,0,0] Assertion <code>srcIndex &lt; srcSelectDimSize</code> failed.<br>
R18_look_at_particular_patch: 1/1000:   0%|                                                    | 0/1844 [00:05&lt;?, ?it/s]<br>
Traceback (most recent call last):<br>
File “main_2.py”, line 171, in <br>
raw_outs = T_net(x)<br>
File “/media/HDD8TB3/swapnil/env/sp_env/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 1102, in _call_impl<br>
return forward_call(*input, **kwargs)<br>
File “/media/HDD8TB3/sourabh/deep_sight_deep_sup_class/gs_front_3_expos_combined/model.py”, line 154, in forward<br>
selected_slices = torch.index_select(feats, 0, indices)<br>
RuntimeError: CUDA error: device-side assert triggered</p>
<p>The code of the model which I am trying to train:</p>
<p>class ResNet_backbone_classifier_grades(nn.Module):<br>
def <strong>init</strong>(self,patch_size=658,num_patches=8,num_feat_maps=512,n_classes=4,model=None):<br>
super().<strong>init</strong>()<br>
self.k = num_feat_maps<br>
self.num_patches = num_patches<br>
self.patch_size= patch_size</p>
<pre><code>    self.model = model  

    ###### classification in grades 
    self.avg_pool_3d = nn.AdaptiveAvgPool3d((64, 2, 2))
    self.fc_phone_class_1 = nn.Linear(256,64)
    self.fc_phone_class_2 = nn.Linear(64,n_classes)
    self.relu = nn.ReLU()

def forward(self,x):
    ###########################################################
    x = x.view(-1,3,self.patch_size,self.patch_size)
    feats, out_1 = self.model(x)

    ###########################################################

    x = torch.softmax(out_1, dim=1)

    a, x = torch.max(x, dim=1)

    all_indices = x.reshape(-1,self.num_patches)

    final_feats = []

    print('All indices in current batch')
    print(all_indices)

    for i in range(len(all_indices)):

        indices = torch.nonzero(all_indices[i]).squeeze()
        print('Indices to pick in current sample: ')
        print(indices)
        if indices.numel() == 0:       
            x = feats[i*self.num_patches:(i+1)*self.num_patches,:,:,:].view(-1,21*21)
            channel_activations, indices = torch.max(x, dim=1)
            topk_values, topk_indices = torch.topk(channel_activations, self.k)
            topk_channels = x[topk_indices]
            x = topk_channels.reshape(-1,self.k,21,21)
            print('no defects')
            print(x.shape)

        else:
            feats = feats[i*self.num_patches:(i+1)*self.num_patches,:,:,:]
            print('Current sample feats dim: ')
            print(feats.shape)
            selected_slices = torch.index_select(feats, 0, indices)
            reshaped_map = selected_slices.view(-1, 21*21)  # Reshape to (-1, 441)
            channel_activations, indices = torch.max(reshaped_map, dim=1)  # Get max activations and indices along dim 1
            topk_values, topk_indices = torch.topk(channel_activations, self.k)
            topk_channels = reshaped_map[topk_indices]
            x = topk_channels.reshape(-1,self.k,21,21)
            print('defects')
            print(x.shape)

        final_feats.append(x)

    final_feats = torch.stack(final_feats,dim=0)
    x = self.avg_pool_3d(final_feats).reshape(final_feats.shape[0],-1)

    x = self.relu(self.fc_phone_class_1(x))
    out_2 = self.fc_phone_class_2(x)

    print(out_2.shape)

    return out_2
</code></pre>
<p>So basically it seems that the issue is related to torch.index_select. As mentioned earlier, I want to pool out the top 512 feature maps (activation-wise) from the variable number of feature maps which are always greater than 512.</p>

Sourabh_Patil: <p>Hey <a class="mention" href="/u/ptrblck">@ptrblck</a>, I solved the issue. There was an issue related to the slicing of the tensor. It was a silly mistake related naming of variables. Thanks for your support.</p>