Illegal memory access

DoKyung_Lim: <p>i have an error after make one loss func</p>
<pre><code class="lang-auto">class Latent_Classifier(nn.Module):
    def __init__(self):
        super(Latent_Classifier, self).__init__()
        
        self.encoder = nn.Sequential(
            nn.Linear(128, 750),
            nn.LeakyReLU(0.2),
            nn.Linear(750, 750),
            nn.Linear(750, 1)
        )
        
    def forward(self,latent_z):
        x1 = self.encoder(latent_z)
        print(x1.size())
        _eps = 1e-15
        loss = -(x1 + _eps).log().mean()-(1 - x1 + _eps).log().mean()
        
        return loss
</code></pre>
<p>i use this func as</p>
<pre><code class="lang-auto">classifier = Latent_Classifier()

f_classifier = classifier(latent_f)
lm_classifier = classifier(latent_l)

loss = 4000*(f_loss + m_loss) + 30 * (f_classifier + lm_classifier) + 2000 * lm_loss

loss.backward()
</code></pre>
<p>in loss.backward() i got an error msg</p>
<p>CUDA error: an illegal memory access was encountered</p>
<p>before using classifier loss, i have no error msg</p>
<p>is an error in function Latent_Classifier?</p>
<p>when i executed it using ‘torch.device(“cpu”)’ not cuda:0 it works well</p>

ptrblck: <p>Could you rerun the script with:</p>
<pre><code class="lang-python">CUDA_LAUNCH_BLOCKING=1 python script.py args
</code></pre>
<p>and post the stack trace here, please?<br>
The illegal memory access might have been created by a previous CUDA operation and your loss could be a red herring.</p>

DoKyung_Lim: <p>it’s my traceback message</p>
<p>Traceback (most recent call last):<br>
File “train2.py”, line 121, in <br>
loss.backward()<br>
File “/home/hhhoh/.local/lib/python3.6/site-packages/torch/tensor.py”, line 18                              5, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph)<br>
File “/home/hhhoh/.local/lib/python3.6/site-packages/torch/autograd/<strong>init</strong>.p                              y”, line 127, in backward<br>
allow_unreachable=True)  # allow_unreachable flag<br>
RuntimeError: CUDA error: an illegal memory access was encountered<br>
Exception raised from copy_kernel_cuda at /pytorch/aten/src/ATen/native/cuda/Cop                              y.cu:200 (most recent call first):<br>
frame <span class="hashtag">#0:</span> c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f130683                              e1e2 in /home/hhhoh/.local/lib/python3.6/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag">#1:</span>  + 0x1e63b08 (0x7f1308b0ab08 in /home/hhhoh/.local/l                              ib/python3.6/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#2:</span>  + 0xc282b9 (0x7f13424cf2b9 in /home/hhhoh/.local/li                              b/python3.6/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#3:</span>  + 0xc25f28 (0x7f13424ccf28 in /home/hhhoh/.local/li                              b/python3.6/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#4:</span> at::native::copy_(at::Tensor&amp;, at::Tensor const&amp;, bool) + 0x44 (0x7f13                              424cf144 in /home/hhhoh/.local/lib/python3.6/site-packages/torch/lib/libtorch_cp                              u.so)<br>
frame <span class="hashtag">#5:</span> at::Tensor::copy_(at::Tensor const&amp;, bool) const + 0x115 (0x7f1342bba0                              95 in /home/hhhoh/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#6:</span>  + 0x37e647e (0x7f134508d47e in /home/hhhoh/.local/l                              ib/python3.6/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#7:</span> at::Tensor::copy_(at::Tensor const&amp;, bool) const + 0x115 (0x7f1342bba0                              95 in /home/hhhoh/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#8:</span> at::native::to(at::Tensor const&amp;, c10::TensorOptions const&amp;, bool, boo                              l, c10::optional<a>c10::MemoryFormat</a>) + 0xb54 (0x7f134270b564 in /home/hhhoh/.loc                              al/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#9:</span>  + 0x128850a (0x7f1342b2f50a in /home/hhhoh/.local/l                              ib/python3.6/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#10:</span>  + 0x2e749da (0x7f134471b9da in /home/hhhoh/.local/                              lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#11:</span>  + 0x10ea412 (0x7f1342991412 in /home/hhhoh/.local/                              lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#12:</span> at::Tensor::to(c10::TensorOptions const&amp;, bool, bool, c10::optional&lt;c                              10::MemoryFormat&gt;) const + 0x146 (0x7f1342bedf56 in /home/hhhoh/.local/lib/pytho                              n3.6/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#13:</span>  + 0x336a970 (0x7f1344c11970 in /home/hhhoh/.local/                              lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#14:</span> torch::autograd::Engine::evaluate_function(std::shared_ptr&lt;torch::aut                              ograd::GraphTask&gt;&amp;, torch::autograd::Node*, torch::autograd::InputBuffer&amp;, std::                              shared_ptr<a>torch::autograd::ReadyQueue</a> const&amp;) + 0x3fd (0x7f1344c173fd in /home                              /hhhoh/.local/lib/python3.6/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#15:</span> torch::autograd::Engine::thread_main(std::shared_ptr&lt;torch::autograd:                              :GraphTask&gt; const&amp;) + 0x451 (0x7f1344c18fa1 in /home/hhhoh/.local/lib/python3.6/                              site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#16:</span> torch::autograd::Engine::thread_init(int, std::shared_ptr&lt;torch::auto                              grad::ReadyQueue&gt; const&amp;, bool) + 0x89 (0x7f1344c11119 in /home/hhhoh/.local/lib                              /python3.6/site-packages/torch/lib/libtorch_cpu.so)<br>
frame <span class="hashtag">#17:</span> torch::autograd::python::PythonEngine::thread_init(int, std::shared_p                              tr<a>torch::autograd::ReadyQueue</a> const&amp;, bool) + 0x4a (0x7f13523b14ba in /home/hh                              hoh/.local/lib/python3.6/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag">#18:</span>  + 0xbd6df (0x7f135350d6df in /usr/lib/x86_64-linux                              -gnu/libstdc++.so.6)<br>
frame <span class="hashtag">#19:</span>  + 0x76db (0x7f13559496db in /lib/x86_64-linux-gnu/                              libpthread.so.0)<br>
frame <span class="hashtag">#20:</span> clone + 0x3f (0x7f1355c82a3f in /lib/x86_64-linux-gnu/libc.so.6)</p>

ptrblck: <p>The illegal memory access was most likely triggered before the copy kernel, so the blocking launch is apparently not working.<br>
Could you post an executable code snippet, which would reproduce this issue?</p>

DoKyung_Lim: <p>Do you mean all code related execution?</p>

ptrblck: <p>No, if possible narrow down the minimal code snippet, which reproduces the error.<br>
I.e. remove all data loading, metric calculation etc., use random inputs and try to isolate the illegal memory access to a few lines.<br>
What’s currently hard to debug is that your code apparently seems to run fine on the CPU and that the blocking launch isn’t properly working in your setup.</p>

DoKyung_Lim: <p>i change classifier to Latent_Classifier().to(device) and error is cleared</p>