Program fails cuda-memcheck

username: neilmehta87
cooked: <p>Hi,</p>
<p>I am new to pytorch and I am trying to write a simple NN code capable of running on GPU. I have attached the code (part of a larger code) which I have written to demonstrate my problem. I am able to compile and run my code without any errors. However, when I try to run it using cuda-memcheck, I get bunch of errors, which all state:<br>
Program hit cudaErrorCudartUnloading (error 4) due to “driver shutting down” on CUDA API call to cudaFree/cudaDeviceSynchronize/cudaEventDestroy etc.<br>
I am using pytorchv1.5.0-gpu, gcc/8.3.0, cuda 10.2.89, python-3.7-anaconda, and running my code on volta v100.</p>
<p>I am attaching my code</p>
<pre><code class="lang-auto">import torch
import numpy as np

class TwoLayerNet(torch.nn.Module):
    def __init__(self, D_in, H, D_out):
        super(TwoLayerNet, self).__init__()
        self.linear1 = torch.nn.Linear(D_in,H)
        self.linear2 = torch.nn.Linear(H,D_out)

    def forward(self, x): 
        h_relu = self.linear1(x).clamp(min=0)
        y_pred = self.linear2(h_relu)
        return y_pred

def run_NN():
    N, D_in, H, D_out, tstep = 64, 1000, 100, 10, 1000
    print("*********************************************")
    print("Start python")

    dtype = torch.float;
    device = torch.device("cuda:0")

    x = torch.randn(N, D_in, dtype=dtype, device=device)
    y = torch.randn(N, D_out, dtype=dtype, device=device)
       
    w1 = torch.randn(D_in, H, device=device, dtype=dtype)
    w2 = torch.randn(H, D_out, device=device, dtype=dtype)  

    model = TwoLayerNet(D_in, H, D_out)
    y_pred = torch.matmul(x,w1)

    print("End python")
    print("*********************************************")

run_NN()
</code></pre>
<p>Could someone please help me?<br>
Thank you</p>
<p>Edit: I tried cuda-memcheck on the examples listed at <a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html" rel="nofollow noopener">Simple NN examples Pytorch</a> and I get the same errors.</p>

username: ptrblck
cooked: <p>The error is raised, as <code>cuda-memcheck</code> is encountering an error, not because it’s finding an issue in the code.<br>
I would recommend to reinstall CUDA and try a new installation of <code>cuda-memcheck</code>.</p>

username: neilmehta87
cooked: <p>Hi ptrblck,<br>
Thank you for responding.<br>
I am using cori NERSC and CUDA is installed centrally on the system. Installed CUDA version is used and tested by others without issue, and cuda-memcheck has been successfully used on multiple other codes.<br>
Could my issue be related to the response posted by <a class="mention" href="/u/ezyang">@ezyang</a> in <a href="https://github.com/pytorch/pytorch/issues/11858#issuecomment-475359500" rel="nofollow noopener">https://github.com/pytorch/pytorch/issues/11858#issuecomment-475359500</a>?</p>
<p>I also tried using binary install of pytorch inside a conda environment and the cuda-memcheck related errors still persist.</p>
<p>Please let me know if I should provide any other information.<br>
Thank you!</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="neilmehta87" data-post="3" data-topic="85997">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/neilmehta87/48/25835_2.png" class="avatar"> neilmehta87:</div>
<blockquote>
<p>Could my issue be related to the response posted by <a class="mention" href="/u/ezyang">@ezyang</a> in <a href="https://github.com/pytorch/pytorch/issues/11858#issuecomment-475359500">https://github.com/pytorch/pytorch/issues/11858#issuecomment-475359500 </a>?</p>
</blockquote>
</aside>
<p>I don’t think so, as I’m using <code>cuda-memcheck</code> regularly and the linked comment is a bit older by now.</p>
<p>You could try to use a docker container and check, if <code>cuda-memcheck</code> is working in this environment.</p>