Apex-0.1-py3.6-linux-x86_64.egg/apex/amp/wrap.py", line 28, in wrapper return orig_fn(*new_args, **kwargs) RuntimeError: CUDA error: an illegal memory access was encountered

iblahimovic_neo: <p>hi everyone,</p>
<p>I met a strange illegal memory access error during evaluation step. It happens randomly. I don’t think there is anything wrong in my evaluation code.<br>
I was training on 4 GPUs(tesla v100), pytorch 1.6, at the same time I was training with mixed precise by using apex.<br>
the error looks like the following error information：</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1c417e211d9bd96a64adf2123970a9073b412788.png" data-download-href="https://discuss.pytorch.org/uploads/default/1c417e211d9bd96a64adf2123970a9073b412788" title="error2"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/c/1c417e211d9bd96a64adf2123970a9073b412788.png" alt="error2" data-base62-sha1="41XGzmIr6swWSzoPmmQSyxcD0Ug" width="690" height="228" data-dominant-color="0B0B0B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error2</span><span class="informations">1112×368 17.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>my evaluation code is</p>
<pre><code class="lang-auto">def evaluate(args, model, features, tag="dev"):
    dev_sampler = torch.utils.data.distributed.DistributedSampler(features)
    dataloader = DataLoader(features, batch_size=args.test_batch_size, num_workers= 
        args.n_gpu, pin_memory=True, shuffle=False, collate_fn=collate_fn, drop_last=True, 
        sampler=dev_sampler)
    preds = []
    labels = []
    tensor_list_pred = [torch.zeros([int(len(dataloader)*args.test_batch_size), args.num_class],   
        dtype= torch.float32, device = args.device) for _ in range(args.n_gpu)]
    tensor_list_label = [torch.zeros([int(len(dataloader)*args.test_batch_size),args.num_class], 
        dtype= torch.float32, device = args.device) for _ in range(args.n_gpu)]
    for batch in dataloader:
        model.eval()

        inputs = {'input_ids': batch[0].to(args.device),
                       'attention_mask': batch[1].to(args.device),
                       'entity_pos': batch[3],
                       'hts': batch[4],
                      }
        label = np.array(batch[2])
        label_t = torch.from_numpy(label)
    
        label_t = label_t.squeeze(1)
        labels.append(label_t.to(args.device))
        with torch.no_grad():
            pred, *_ = model(**inputs)
            pred = pred.cpu().numpy()
            pred[np.isnan(pred)] = 0
            pred = torch.from_numpy(pred)
            preds.append(pred.to(args.device))
    
        
    label_s = torch.cat(labels, axis=0)
    pred_s = torch.cat(preds, axis=0)

    dist.all_gather(tensor_list_pred, pred_s)
    dist.all_gather(tensor_list_label, label_s)

    labels = torch.cat(tensor_list_label, axis=0)
    preds = torch.cat(tensor_list_pred, axis=0)

    labels_c = labels.cpu()
    preds_c = preds.cpu()

    r,c = preds_c.size()
    _ , index = torch.topk(preds_c, c, dim=1)
    pred_nr = index[:,0].numpy()
    _ , index_label = torch.topk(labels_c, c, dim=1)
    y_true = index_label[:,0].numpy()
    f1_macro = f1_score(y_true, pred_nr, average='macro')
    f1_micro = f1_score(y_true, pred_nr, average='micro')
    f1_weighted = f1_score(y_true, pred_nr, average='weighted')
    output = {
        tag + "_F1_micro": f1_micro * 100,
        tag + "_F1_macro": f1_macro * 100,
        tag + "_F1_weighted": f1_weighted * 100,
        tag + "_class_report": classification_report(y_true, pred_nr),
    }
    return f1_weighted, output
</code></pre>
<p>I have been stucked here for several days. who can tell me which reason causes this error? Could you give me some suggestions for fixing this problem.</p>
<p>Thanks in advance</p>

ptrblck: <p>I don’t see where <code>apex</code> is used, but note that we recommend to use the native mixed-precision implementation via <a href="https://pytorch.org/docs/stable/amp.html"><code>torch.cuda.amp</code></a> as well as the native <code>DistributedDataParallel</code> implementation.</p>

iblahimovic_neo: <p>instead of apex I have used torch.cuda.amp for mixed precision implementation. but same error information has been shown. because the variablel label is on CPU. In order to accumulate<br>
the label value from 4 different processes I have used the torch API torch.distributed.all_gather. But the API must run on cuda. And then I transfered the label value from CPU to GPU. After getting all of label value, I transfered the label value from GPU to CPU for calculating the metric value(f1 score) by using  sklearn.metrics.f1_score. But when the code runs to this row labels_c = labels.cpu(). the error information has been shown again.<br>
I dont know what caused this.</p>

ptrblck: <p>Could you post an executable code snippet so that we could reproduce this error, please?</p>