Error in `python': double free or corruption (fasttop)

username: jaobalao
cooked: <p><strong>Issue:</strong><br>
I am getting a segfault error when I train an RNN model using num_workers&gt;0 on the Dataloader. The data is being loaded from a kyoto cabinet file, using a custom collate function.</p>
<p><strong>Question:</strong><br>
What can I do to solve this problem?</p>
<p><strong>Trying to isolate the issue:</strong><br>
If I comment the code related to the loss function/optimization (nn.BCELoss, backward, adam) inside the train batch generator loop, the code works fine for several epochs. If I train with num_workers=0 on the Dataloader, there are no issues either and I can also train for several epochs (very slow though). When I uncomment the loss/optimization code, it usually works for the first half of the mini-batches but then I get the memory error on the second half - always on the first epoch.</p>
<p><strong>Environment:</strong></p>
<p>PyTorch Version: 1.1.0<br>
conda install pytorch torchvision cudatoolkit=10.0 -c pytorch</p>
<p>Anaconda version: Anaconda3-2019.03-Linux-x86_64.sh</p>
<p>$ python --version<br>
Python 3.7.3</p>
<p>$ cat /usr/local/cuda/version.txt<br>
CUDA Version 10.1.168</p>
<p>$ cat /etc/os-release<br>
NAME=“CentOS Linux”<br>
VERSION=“7 (Core)”<br>
ID=“centos”<br>
ID_LIKE=“rhel fedora”<br>
VERSION_ID=“7”<br>
PRETTY_NAME=“CentOS Linux 7 (Core)”<br>
ANSI_COLOR=“0;31”<br>
CPE_NAME=“cpe:/o:centos:centos:7”<br>
HOME_URL=“<a href="https://www.centos.org/" rel="nofollow noopener">https://www.centos.org/</a>”<br>
BUG_REPORT_URL=“<a href="https://bugs.centos.org/" rel="nofollow noopener">https://bugs.centos.org/</a>”</p>
<p>CENTOS_MANTISBT_PROJECT=“CentOS-7”<br>
CENTOS_MANTISBT_PROJECT_VERSION=“7”<br>
REDHAT_SUPPORT_PRODUCT=“centos”<br>
REDHAT_SUPPORT_PRODUCT_VERSION=“7”</p>
<p>$ gcc --version<br>
gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-36)<br>
Copyright © 2015 Free Software Foundation, Inc.<br>
This is free software; see the source for copying conditions.  There is NO<br>
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.</p>
<p><strong>Error log:</strong><br>
*** Error in `python’: double free or corruption (fasttop): 0x00007f39fc04af40 *** time: 0:12:22<br>
======= Backtrace: =========<br>
/usr/lib64/libc.so.6(+0x81609)[0x7f3b58f82609]<br>
/usr/lib64/libcuda.so.1(+0x2011f2)[0x7f3afb3c71f2]<br>
/usr/lib64/libcuda.so.1(+0x10abc2)[0x7f3afb2d0bc2]<br>
/usr/lib64/libcuda.so.1(+0x10ac39)[0x7f3afb2d0c39]<br>
/usr/lib64/libcuda.so.1(cuStreamCreate+0x5b)[0x7f3afb41635b]<br>
/opt/mapr/tools/python/anaconda3/lib/python3.7/site-packages/torch/lib/…/…/…/…/libcudart.so.10.0(+0xffa2)[0x7f3b430ecfa2]<br>
/opt/mapr/tools/python/anaconda3/lib/python3.7/site-packages/torch/lib/…/…/…/…/libcudart.so.10.0(cudaStreamCreate+0x64)[0x7f3b43126874]<br>
/opt/mapr/tools/python/anaconda3/lib/python3.7/site-packages/torch/lib/libcaffe2_gpu.so(_ZN17RNNBackwardFilterIfffE4initEP12cudnnContextP14cudnnRNNStructi11PerfOptions+0x3b0)[0x7f3b16a725c0]<br>
/opt/mapr/tools/python/anaconda3/lib/python3.7/site-packages/torch/lib/libcaffe2_gpu.so(_Z24RNN_WGRAD_LaunchTemplateIfffE13cudnnStatus_tP12cudnnContextP14cudnnRNNStructiPKP17cudnnTensorStructPKvSA_SA_SA_mPvSA_m11PerfOptions+0x80)[0x7f3b16a76270]<br>
/opt/mapr/tools/python/anaconda3/lib/python3.7/site-packages/torch/lib/libcaffe2_gpu.so(cudnnRNNBackwardWeights+0xf02)[0x7f3b16a71602]<br>
/opt/mapr/tools/python/anaconda3/lib/python3.7/site-packages/torch/lib/libcaffe2_gpu.so(<em>ZN2at6native26_cudnn_rnn_backward_weightERKNS_6TensorEN3c108ArrayRefIS1_EElS3_S3_S3_S3_lllbdbbNS5_IlEES3_S3</em>+0xc0b)[0x7f3b142accbb]<br>
/opt/mapr/tools/python/anaconda3/lib/python3.7/site-packages/torch/lib/libcaffe2_gpu.so(_ZN2at6native19_cudnn_rnn_backwardERKNS_6TensorEN3c108ArrayRefIS1_EElS3_S3_S3_S3_S3_S3_S3_lllbdbbNS5_IlEES3_S3_St5arrayIbLm4EE+0x2f6)[0x7f3b142b3106]<br>
/opt/mapr/tools/python/anaconda3/lib/python3.7/site-packages/torch/lib/libcaffe2_gpu.so(_ZNK2at8CUDAType19_cudnn_rnn_backwardERKNS_6TensorEN3c108ArrayRefIS1_EElS3_S3_S3_S3_S3_S3_S3_lllbdbbNS5_IlEES3_S3_St5arrayIbLm4EE+0x178)[0x7f3b1438a7f8]<br>
/opt/mapr/tools/python/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch.so.1(_ZNK5torch8autograd12VariableType19_cudnn_rnn_backwardERKN2at6TensorEN3c108ArrayRefIS3_EElS5_S5_S5_S5_S5_S5_S5_lllbdbbNS7_IlEES5_S5_St5arrayIbLm4EE+0x1101)[0x7f3b12421f81]<br>
/opt/mapr/tools/python/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch.so.1(_ZN5torch8autograd9generated16CudnnRnnBackward5applyEOSt6vectorINS0_8VariableESaIS4_EE+0x6d4)[0x7f3b121c9f04]<br>
/opt/mapr/tools/python/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch.so.1(+0x307622)[0x7f3b121ab622]<br>
/opt/mapr/tools/python/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch.so.1(_ZN5torch8autograd6Engine17evaluate_functionERNS0_12FunctionTaskE+0x385)[0x7f3b121a4745]<br>
/opt/mapr/tools/python/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch.so.1(_ZN5torch8autograd6Engine11thread_mainEPNS0_9GraphTaskE+0xc0)[0x7f3b121a6740]<br>
/opt/mapr/tools/python/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch.so.1(_ZN5torch8autograd6Engine11thread_initEi+0x2b0)[0x7f3b121a39e0]<br>
/opt/mapr/tools/python/anaconda3/lib/python3.7/site-packages/torch/lib/libtorch_python.so(_ZN5torch8autograd6python12PythonEngine11thread_initEi+0x2a)[0x7f3b3efc228a]<br>
/opt/mapr/tools/python/anaconda3/lib/libstdc++.so.6(+0xb8678)[0x7f3b57e88678]<br>
/usr/lib64/libpthread.so.0(+0x7dd5)[0x7f3b592d5dd5]<br>
/usr/lib64/libc.so.6(clone+0x6d)[0x7f3b58fff02d]<br>
======= Memory map: ========<br>
200000000-200200000 rw-s 00000000 00:05 21694                            /dev/nvidiactl<br>
200200000-200600000 —p 00000000 00:00 0<br>
200600000-200800000 rw-s 00000000 00:05 21694                            /dev/nvidiactl<br>
200800000-200a00000 rw-s 00000000 00:05 27538                            /dev/nvidia0<br>
200a00000-206200000 rw-s 00000000 00:05 21694                            /dev/nvidiactl<br>
206200000-206400000 rw-s 00000000 00:05 27538                            /dev/nvidia0<br>
206400000-207400000 —p 00000000 00:00 0<br>
207400000-207600000 rw-s 00000000 00:05 21694                            /dev/nvidiactl<br>
207600000-207800000 rw-s 00000000 00:05 21694                            /dev/nvidiactl<br>
207800000-207a00000 rw-s 207800000 00:05 19518                           /dev/nvidia-uvm<br>
207a00000-207c00000 —p 00000000 00:00 0<br>
207c00000-207e00000 rw-s 00000000 00:05 21694                            /dev/nvidiactl<br>
207e00000-208000000 rw-s 00000000 00:04 186466                           /dev/zero (deleted)<br>
208000000-300200000 —p 00000000 00:00 0<br>
10000000000-10004000000 —p 00000000 00:00 0<br>
7f390c000000-7f390c021000 rw-p 00000000 00:00 0<br>
7f390c021000-7f3910000000 —p 00000000 00:00 0<br>
7f3914000000-7f39f8000000 —p 00000000 00:00 0<br>
7f39f8000000-7f39f8021000 rw-p 00000000 00:00 0<br>
7f39f8021000-7f39fc000000 —p 00000000 00:00 0<br>
7f39fc000000-7f39fc107000 rw-p 00000000 00:00 0<br>
7f39fc107000-7f3a00000000 —p 00000000 00:00 0<br>
7f3a00000000-7f3a30000000 —p 00000000 00:00 0<br>
7f3a30000000-7f3a30021000 rw-p 00000000 00:00 0<br>
7f3a30021000-7f3a34000000 —p 00000000 00:00 0<br>
7f3a34000000-7f3a34021000 rw-p 00000000 00:00 0<br>
7f3a34021000-7f3a38000000 —p 00000000 00:00 0<br>
7f3a38000000-7f3a38021000 rw-p 00000000 00:00 0<br>
7f3a38021000-7f3a3c000000 —p 00000000 00:00 0<br>
7f3a3c000000-7f3a3c021000 rw-p 00000000 00:00 0<br>
7f3a3c021000-7f3a40000000 —p 00000000 00:00 0<br>
7f3a40000000-7f3a40021000 rw-p 00000000 00:00 0<br>
7f3a40021000-7f3a44000000 —p 00000000 00:00 0<br>
7f3a447f9000-7f3a447fa000 —p 00000000 00:00 0<br>
7f3a447fa000-7f3a44ffa000 rw-p 00000000 00:00 0                          [stack:17157]<br>
7f3a44ffa000-7f3a44ffb000 —p 00000000 00:00 0<br>
7f3a44ffb000-7f3a457fb000 rw-p 00000000 00:00 0                          [stack:17156]<br>
7f3a457fb000-7f3a457fc000 —p 00000000 00:00 0<br>
7f3a457fc000-7f3a45ffc000 rw-p 00000000 00:00 0                          [stack:17113]<br>
7f3a45ffc000-7f3a45ffd000 —p 00000000 00:00 0<br>
7f3a45ffd000-7f3a467fd000 rw-p 00000000 00:00 0                          [stack:17112]<br>
7f3a467fd000-7f3a467fe000 —p 00000000 00:00 0<br>
7f3a467fe000-7f3a46ffe000 rw-p 00000000 00:00 0                          [stack:17111]<br>
7f3a46ffe000-7f3a46fff000 —p 00000000 00:00 0<br>
7f3a46fff000-7f3a477ff000 rw-p 00000000 00:00 0                          [stack:17110]<br>
7f3a477ff000-7f3a47800000 —p 00000000 00:00 0<br>
7f3a47800000-7f3a48000000 rw-p 00000000 00:00 0                          [stack:17109]<br>
7f3a48000000-7f3a48021000 rw-p 00000000 00:00 0<br>
7f3a48021000-7f3a4c000000 —p 00000000 00:00 0<br>
7f3a4c000000-7f3a4c021000 rw-p 00000000 00:00 0<br>
7f3a4c021000-7f3a50000000 —p 00000000 00:00 0<br>
7f3a50000000-7f3a50021000 rw-p 00000000 00:00 0<br>
7f3a50021000-7f3a54000000 —p 00000000 00:00 0<br>
7f3a54000000-7f3a54021000 rw-p 00000000 00:00 0<br>
7f3a54021000-7f3a58000000 —p 00000000 00:00 0<br>
7f3a58000000-7f3a58021000 rw-p 00000000 00:00 0<br>
7f3a58021000-7f3a5c000000 —p 00000000 00:00 0<br>
7f3a5c000000-7f3a5c021000 rw-p 00000000 00:00 0<br>
7f3a5c021000-7f3a60000000 —p 00000000 00:00 0<br>
7f3a60000000-7f3a60021000 rw-p 00000000 00:00 0<br>
7f3a60021000-7f3a64000000 —p 00000000 00:00 0<br>
7f3a64000000-7f3a64021000 rw-p 00000000 00:00 0<br>
7f3a64021000-7f3a68000000 —p 00000000 00:00 0<br>
7f3a68000000-7f3a68021000 rw-p 00000000 00:00 0<br>
7f3a68021000-7f3a6c000000 —p 00000000 00:00 0<br>
7f3a6c000000-7f3a6c021000 rw-p 00000000 00:00 0<br>
7f3a6c021000-7f3a70000000 —p 00000000 00:00 0<br>
7f3a70000000-7f3a70021000 rw-p 00000000 00:00 0<br>
7f3a70021000-7f3a74000000 —p 00000000 00:00 0<br>
7f3a74000000-7f3a74021000 rw-p 00000000 00:00 0<br>
7f3a74021000-7f3a78000000 —p 00000000 00:00 0<br>
7f3a78000000-7f3a78021000 rw-p 00000000 00:00 0<br>
7f3a78021000-7f3a7c000000 —p 00000000 00:00 0<br>
7f3a7c000000-7f3a7c021000 rw-p 00000000 00:00 0<br>
7f3a7c021000-7f3a80000000 —p 00000000 00:00 0<br>
7f3a80000000-7f3a80021000 rw-p 00000000 00:00 0<br>
7f3a80021000-7f3a84000000 —p 00000000 00:00 0<br>
7f3a847fd000-7f3a847fe000 —p 00000000 00:00 0<br>
7f3a847fe000-7f3a84ffe000 rw-p 00000000 00:00 0                          [stack:17108]<br>
7f3a84ffe000-7f3a84fff000 —p 00000000 00:00 0<br>
7f3a84fff000-7f3a857ff000 rw-p 00000000 00:00 0                          [stack:17107]<br>
7f3a857ff000-7f3a85800000 —p 00000000 00:00 0<br>
7f3a85800000-7f3a86000000 rw-p 00000000 00:00 0                          [stack:17106]<br>
7f3a86000000-7f3aa0000000 —p 00000000 00:00 0<br>
7f3aa03ca000-7f3aa040a000 rw-p 00000000 00:00 0<br>
7f3aa040a000-7f3aa040b000 —p 00000000 00:00 0<br>
7f3aa040b000-7f3aa0c0b000 rw-p 00000000 00:00 0                          [stack:17105]<br>
7f3aa0c0b000-7f3aa0c0c000 —p 00000000 00:00 0<br>
7f3aa0c0c000-7f3aa3a06000 rw-p 00000000 00:00 0                          [stack:17104]<br>
7f3aa3ffc000-7f3aa3ffd000 —p 00000000 00:00 0<br>
7f3aa3ffd000-7f3aa47fd000 rw-p 00000000 00:00 0                          [stack:17103]<br>
7f3aa47fd000-7f3aa47fe000 —p 00000000 00:00 0<br>
7f3aa47fe000-7f3aa4ffe000 rw-p 00000000 00:00 0                          [stack:17102]<br>
7f3aa4ffe000-7f3aa4fff000 —p 00000000 00:00 0<br>
7f3aa4fff000-7f3aa57ff000 rw-p 00000000 00:00 0                          [stack:17101]<br>
7f3aa57ff000-7f3aa5800000 —p 00000000 00:00 0<br>
7f3aa5800000-7f3aa6000000 rw-p 00000000 00:00 0                          [stack:17100]<br>
7f3aa6000000-7f3ab1c00000 —p 00000000 00:00 0<br>
7f3ab1c00000-7f3ab1e00000 rw-s 00000000 00:04 189555                     /dev/zero (deleted)<br>
7f3ab1e00000-7f3ab9000000 —p 00000000 00:00 0<br>
7f3ab9000000-7f3ab9200000 rw-s 00000000 00:04 186469                     /dev/zero (deleted)<br>
7f3ab9200000-7f3ace400000 —p 00000000 00:00 0<br>
7f3ace400000-7f3ace600000 rw-s 00000000 00:04 186464                     /dev/zero (Aborted</p>

username: ecolss
cooked: <p>Did you get a solution?</p>