Free(): invalid pointer Aborted (core dumped) on loss.backward()

Akib_Sadmanee: <p>I am trying to build a next word prediction model with bidirectional LSTM using<br>
<a href="https://pytorch.org/docs/stable/generated/torch.nn.AdaptiveLogSoftmaxWithLoss.html" rel="noopener nofollow ugc">AdaptiveLogSoftmaxWithLoss</a> and Adam optimizer.  However, after processing 40 batches of data, I’m getting <code>free(): invalid pointer, Aborted (core dumped)</code>. The code runs fine after removing the loss.backward() function. So, I decided that the error had to do something with that line. I ran the code in gdp and here’s the trace.</p>
<pre><code class="lang-auto">#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff7de5859 in __GI_abort () at abort.c:79
#2  0x00007ffff7e503ee in __libc_message (action=action@entry=do_abort, 
    fmt=fmt@entry=0x7ffff7f7a285 "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#3  0x00007ffff7e5847c in malloc_printerr (
    str=str@entry=0x7ffff7f7c9f8 "malloc(): memory corruption (fast)")
    at malloc.c:5347
#4  0x00007ffff7e5b5bc in _int_malloc (av=av@entry=0x7ffe7c000020, 
    bytes=bytes@entry=32) at malloc.c:3594
#5  0x00007ffff7e5ed15 in __libc_calloc (n=&lt;optimized out&gt;, 
    elem_size=&lt;optimized out&gt;) at malloc.c:3428
#6  0x00007fff338064f6 in cudnnHostCalloc(unsigned long, unsigned long) ()
   from /home/akib/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cpp.so
#7  0x00007fff33b77d0e in RNNBackwardData&lt;float, float, float&gt;::init(cudnnContext*, cudnnRNNStruct*, int, PerfOptions) ()
   from /home/akib/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cpp.so
#8  0x00007fff32901097 in cudnnStatus_t RNN_DGRAD_LaunchTemplate&lt;float, float, float, true&gt;(cudnnContext*, cudnnRNNStruct*, int, cudnnTensorStruct* const*, void const*, void const*, void const*, void const*, void const*, void const*, cudnnTensorStruct* const*, void*, void*, void*, void const*, void*, void*, PerfOptions, bool) ()
--Type &lt;RET&gt; for more, q to quit, c to continue without paging--c
   from /home/akib/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cpp.so
#9  0x00007fff33b57b75 in cudnnRNNBackwardData () from /home/akib/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cpp.so
#10 0x00007fff32d175a8 in at::native::_cudnn_rnn_backward_input(at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, long, long, long, long, bool, double, bool, bool, c10::ArrayRef&lt;long&gt;, at::Tensor const&amp;, at::Tensor const&amp;, std::array&lt;bool, 3ul&gt;) () from /home/akib/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cpp.so
#11 0x00007fff32d1acf0 in at::native::_cudnn_rnn_backward(at::Tensor const&amp;, c10::ArrayRef&lt;at::Tensor&gt;, long, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, long, long, long, long, bool, double, bool, bool, c10::ArrayRef&lt;long&gt;, at::Tensor const&amp;, at::Tensor const&amp;, std::array&lt;bool, 4ul&gt;) () from /home/akib/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cpp.so
#12 0x00007fff9506d854 in c10::impl::wrap_kernel_functor_unboxed_&lt;c10::impl::detail::WrapFunctionIntoFunctor_&lt;c10::CompileTimeFunctionPointer&lt;std::tuple&lt;at::Tensor, at::Tensor, at::Tensor, std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt; &gt; (at::Tensor const&amp;, c10::ArrayRef&lt;at::Tensor&gt;, long, at::Tensor const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, long, long, long, long, bool, double, bool, bool, c10::ArrayRef&lt;long&gt;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, std::array&lt;bool, 4ul&gt;), &amp;c10::impl::detail::with_explicit_optional_tensors_&lt;std::tuple&lt;at::Tensor, at::Tensor, at::Tensor, std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt; &gt; (at::Tensor const&amp;, c10::ArrayRef&lt;at::Tensor&gt;, long, at::Tensor const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, long, long, long, long, bool, double, bool, bool, c10::ArrayRef&lt;long&gt;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, std::array&lt;bool, 4ul&gt;), std::tuple&lt;at::Tensor, at::Tensor, at::Tensor, std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt; &gt; (at::Tensor const&amp;, c10::ArrayRef&lt;at::Tensor&gt;, long, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, long, long, long, long, bool, double, bool, bool, c10::ArrayRef&lt;long&gt;, at::Tensor const&amp;, at::Tensor const&amp;, std::array&lt;bool, 4ul&gt;), c10::CompileTimeFunctionPointer&lt;std::tuple&lt;at::Tensor, at::Tensor, at::Tensor, std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt; &gt; (at::Tensor const&amp;, c10::ArrayRef&lt;at::Tensor&gt;, long, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, at::Tensor const&amp;, long, long, long, long, bool, double, bool, bool, c10::ArrayRef&lt;long&gt;, at::Tensor const&amp;, at::Tensor const&amp;, std::array&lt;bool, 4ul&gt;), &amp;at::(anonymous namespace)::(anonymous namespace)::wrapper__cudnn_rnn_backward&gt; &gt;::wrapper&gt;, std::tuple&lt;at::Tensor, at::Tensor, at::Tensor, std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt; &gt;, c10::guts::typelist::typelist&lt;at::Tensor const&amp;, c10::ArrayRef&lt;at::Tensor&gt;, long, at::Tensor const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, long, long, long, long, bool, double, bool, bool, c10::ArrayRef&lt;long&gt;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, std::array&lt;bool, 4ul&gt; &gt; &gt;, std::tuple&lt;at::Tensor, at::Tensor, at::Tensor, std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt; &gt; (at::Tensor const&amp;, c10::ArrayRef&lt;at::Tensor&gt;, long, at::Tensor const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, long, long, long, long, bool, double, bool, bool, c10::ArrayRef&lt;long&gt;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, std::array&lt;bool, 4ul&gt;)&gt;::call(c10::OperatorKernel*, at::Tensor const&amp;, c10::ArrayRef&lt;at::Tensor&gt;, long, at::Tensor const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, long, long, long, long, bool, double, bool, bool, c10::ArrayRef&lt;long&gt;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, std::array&lt;bool, 4ul&gt;) () from /home/akib/.local/lib/python3.8/site-packages/torch/lib/libtorch_cuda_cu.so
#13 0x00007fff834b6687 in at::_cudnn_rnn_backward(at::Tensor const&amp;, c10::ArrayRef&lt;at::Tensor&gt;, long, at::Tensor const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, long, long, long, long, bool, double, bool, bool, c10::ArrayRef&lt;long&gt;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, std::array&lt;bool, 4ul&gt;) () from /home/akib/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#14 0x00007fff84cc3c59 in torch::autograd::VariableType::(anonymous namespace)::_cudnn_rnn_backward(at::Tensor const&amp;, c10::ArrayRef&lt;at::Tensor&gt;, long, at::Tensor const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, long, long, long, long, bool, double, bool, bool, c10::ArrayRef&lt;long&gt;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, std::array&lt;bool, 4ul&gt;) () from /home/akib/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#15 0x00007fff84cc4582 in c10::impl::wrap_kernel_functor_unboxed_&lt;c10::impl::detail::WrapFunctionIntoFunctor_&lt;c10::CompileTimeFunctionPointer&lt;std::tuple&lt;at::Tensor, at::Tensor, at::Tensor, std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt; &gt; (at::Tensor const&amp;, c10::ArrayRef&lt;at::Tensor&gt;, long, at::Tensor const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, long, long, long, long, bool, double, bool, bool, c10::ArrayRef&lt;long&gt;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, std::array&lt;bool, 4ul&gt;), &amp;torch::autograd::VariableType::(anonymous namespace)::_cudnn_rnn_backward&gt;, std::tuple&lt;at::Tensor, at::Tensor, at::Tensor, std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt; &gt;, c10::guts::typelist::typelist&lt;at::Tensor const&amp;, c10::ArrayRef&lt;at::Tensor&gt;, long, at::Tensor const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, long, long, long, long, bool, double, bool, bool, c10::ArrayRef&lt;long&gt;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, std::array&lt;bool, 4ul&gt; &gt; &gt;, std::tuple&lt;at::Tensor, at::Tensor, at::Tensor, std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt; &gt; (at::Tensor const&amp;, c10::ArrayRef&lt;at::Tensor&gt;, long, at::Tensor const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, long, long, long, long, bool, double, bool, bool, c10::ArrayRef&lt;long&gt;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, std::array&lt;bool, 4ul&gt;)&gt;::call(c10::OperatorKernel*, at::Tensor const&amp;, c10::ArrayRef&lt;at::Tensor&gt;, long, at::Tensor const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, long, long, long, long, bool, double, bool, bool, c10::ArrayRef&lt;long&gt;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, std::array&lt;bool, 4ul&gt;) () from /home/akib/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#16 0x00007fff834b6687 in at::_cudnn_rnn_backward(at::Tensor const&amp;, c10::ArrayRef&lt;at::Tensor&gt;, long, at::Tensor const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, c10::optional&lt;at::Tensor&gt; const&amp;, long, long, long, long, bool, double, bool, bool, c10::ArrayRef&lt;long&gt;, c10::optional&lt;at::Tensor&gt; const&amp;, at::Tensor const&amp;, std::array&lt;bool, 4ul&gt;) () from /home/akib/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#17 0x00007fff84c08026 in torch::autograd::generated::CudnnRnnBackward::apply(std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt;&amp;&amp;) () from /home/akib/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#18 0x00007fff85275771 in torch::autograd::Node::operator()(std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt;&amp;&amp;) () from /home/akib/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#19 0x00007fff8527157b in torch::autograd::Engine::evaluate_function(std::shared_ptr&lt;torch::autograd::GraphTask&gt;&amp;, torch::autograd::Node*, torch::autograd::InputBuffer&amp;, std::shared_ptr&lt;torch::autograd::ReadyQueue&gt; const&amp;) () from /home/akib/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#20 0x00007fff8527219f in torch::autograd::Engine::thread_main(std::shared_ptr&lt;torch::autograd::GraphTask&gt; const&amp;) () from /home/akib/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#21 0x00007fff85269979 in torch::autograd::Engine::thread_init(int, std::shared_ptr&lt;torch::autograd::ReadyQueue&gt; const&amp;, bool) () from /home/akib/.local/lib/python3.8/site-packages/torch/lib/libtorch_cpu.so
#22 0x00007ffff59ed293 in torch::autograd::python::PythonEngine::thread_init(int, std::shared_ptr&lt;torch::autograd::ReadyQueue&gt; const&amp;, bool) () from /home/akib/.local/lib/python3.8/site-packages/torch/lib/libtorch_python.so
#23 0x00007ffff6a04d84 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#24 0x00007ffff7da6609 in start_thread (arg=&lt;optimized out&gt;) at pthread_create.c:477
#25 0x00007ffff7ee2293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95
</code></pre>
<p>I searched for some solutions and tried reinstalling pytorch and adding gc.collect() before calling backward(). I’m adding a snippet of my code for further reference</p>
<pre><code class="lang-auto">class BLSTM(torch.nn.Module):
    def __init__(self, emb_size, hidden_size, num_layers, vocab_size, cutoffs):
        super(BLSTM, self).__init__()
        self.hidden_size = hidden_size
        self.num_layers = num_layers
        self.blstm = torch.nn.LSTM(emb_size, hidden_size, num_layers, batch_first=True, bidirectional=True)
        self.adasoft = torch.nn.AdaptiveLogSoftmaxWithLoss(hidden_size*2, vocab_size, cutoffs)
    
    def forward(self, x, targets):
        # Set initial states
        h0 = torch.zeros(self.num_layers*2, x.size(0), self.hidden_size).to(device)
        c0 = torch.zeros(self.num_layers*2, x.size(0), self.hidden_size).to(device)

        # Forward propagate LSTM
        embed = torch.nn.Embedding(vocab_size, emb_size).to(device)
        out, _ = self.blstm(embed(x), (h0, c0))  # out: tensor of shape (batch_size, seq_length, hidden_size*2)
        
        # Decode the hidden state of the last time step
        out = self.adasoft(out[:, -1, :], targets)
        
        return out
</code></pre>
<pre><code class="lang-auto">optimizer.zero_grad()
outputs = model(input_seq, target_seq)
outputs.loss.backward()
optimizer.step()
</code></pre>
<p>I am using CUDA11.2 in Ubuntu20.04 machine.</p>

ptrblck: <p>It seems that a <code>cudnnHostCalloc</code> call is failing. How did you install PyTorch and which cudnn version are you using?<br>
Since you mentioned CUDA11.2, did you build PyTorch from source or are you using the binaries?<br>
Also, which GPU are you using?</p>

Akib_Sadmanee: <pre><code class="lang-auto">nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2019 NVIDIA Corporation
Built on Sun_Jul_28_19:07:16_PDT_2019
Cuda compilation tools, release 10.1, V10.1.243
</code></pre>
<p>I installed pytorch through pip using the command created by <a href="http://pytorch.org" rel="noopener nofollow ugc">pytorch.org</a></p>
<pre><code class="lang-auto">pip install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio==0.8.1 -f https://download.pytorch.org/whl/torch_stable.html
</code></pre>
<p>And my GPU is NVIDIA Geforce GTX 1050Ti</p>
<p>However, I’m not sure how to check CuDNN version. This is what I see after running the installation file- “libcudnn8_8.1.1.33-1+cuda11.2_amd64.deb”<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/8/c86558d6c0644446f09c64baab88d1054ca93a53.png" alt="Screenshot from 2021-03-30 15-10-01" data-base62-sha1="sAMD0ezHQSg5kAluZyCgTHjBy9R" width="660" height="95"><br>
I have never set up a GPU device before. So I’m not sure what I’m doing to be honest.</p>

ptrblck: <p>Thanks for the update.<br>
Based on your description you’ve installed the pip wheels, which ship with their CUDA runtime as well as cudnn (8.0.5). Your local CUDA toolkit and cudnn won’t be used in this case.</p>
<p>Since you are using a local CUDA10.1 installation, I guess your NVIDIA driver might be a bit older?<br>
Could you check it via <code>nvidia-smi</code> and make sure it’s new enough to run the CUDA11.1 runtime from the pip wheels? (<a href="https://docs.nvidia.com/deploy/cuda-compatibility/index.html#:~:text=The%20CUDA%20Driver%20API%20thus,is%20in%20one%20direction%3A%20backwards.">Table1</a> gives you an overview of the driver requirements)</p>

Akib_Sadmanee: <p>Thanks for the heads up. Here is the nvidia-smi result. The driver version is 460 which should be able to run CUDA 11.2 according to the Table 1 you mentioned.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/9/69451505e51fc3ffa64783f2e24dac738641cb14.png" data-download-href="https://discuss.pytorch.org/uploads/default/69451505e51fc3ffa64783f2e24dac738641cb14" title="Screenshot from 2021-03-31 12-49-48"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/9/69451505e51fc3ffa64783f2e24dac738641cb14.png" alt="Screenshot from 2021-03-31 12-49-48" data-base62-sha1="f1g8OswnInLU3Sw4W19IlpPtjTe" width="690" height="379" data-dominant-color="000F00"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2021-03-31 12-49-48</span><span class="informations">716×394 14.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>