During the training of the first epoch, it's killed

lin4mation: <p>Hi,</p>
<p>During the training of the first epoch, the program got “killed” after 957/2354 batches and there was no other message.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/9/19ef2727adf55f8ec6c9058e5921693303632235.png" data-download-href="https://discuss.pytorch.org/uploads/default/19ef2727adf55f8ec6c9058e5921693303632235" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/19ef2727adf55f8ec6c9058e5921693303632235_2_690x39.png" alt="image" data-base62-sha1="3HqjOvGediBSU7w8jThlLoZK7uR" width="690" height="39" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/19ef2727adf55f8ec6c9058e5921693303632235_2_690x39.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/9/19ef2727adf55f8ec6c9058e5921693303632235_2_1035x58.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/9/19ef2727adf55f8ec6c9058e5921693303632235.png 2x" data-dominant-color="1E1E1E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1208×69 7.66 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The dataloader was using the default <code>num_workers=0</code>.<br>
The code ran well when I ran through a smaller dataset.<br>
I’m using GPU.<br>
I did set <code>optimizer.zero_grad()</code> at the beginning of each batch within the dataloader loop.<br>
Is it the issue of GPU memory?<br>
Do I need to set <code>torch.cuda.empty_cache()</code> after each batch?</p>
<p>Here’s the versions I am using.</p>
<pre><code class="lang-auto"># Name                    Version                   Build  Channel
pytorch                   1.2.0           py3.6_cuda9.2.148_cudnn7.6.2_0    pytorch
torchvision               0.4.0                 py36_cu92    pytorch
</code></pre>
<p>Please let me know if you have any suggestions. Thank you.</p>

albanD: <p>Hi,</p>
<p>Do you run out of RAM during your training? This message usually comes from the system running out of memory and starting killing process (or you doing <code>kill -9 PID</code>).</p>

lin4mation: <p>Hi,<br>
I don’t think that’s the case because the RAM has 1T total so I think it’s unlikely to run out of RAM. I didn’t start any killing process either.</p>

albanD: <p>Is there some limitations on how much RAM a process of your user is allowed to use on your machine (from cgroup on linux or similar mechanisms)?</p>

lin4mation: <p>No. I am the only one user and there is no memory limit for a process.</p>

albanD: <p>How do you check for the absence of limits on processes?</p>

lin4mation: <p>Does this mean no limit of max of memory?<br>
$ <code>ulimit -a</code></p>
<pre><code class="lang-auto">core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 4127735
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4127735
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
</code></pre>

albanD: <p>You can try and follow the advices <a href="https://stackoverflow.com/questions/726690/what-killed-my-process-and-why" rel="nofollow noopener">here</a> to try and pinpoint the root cause for this.</p>

imaginist: <p>Hi, I got the same problem as you. Have you solved this issue now?</p>

Saida2020: <p>Hello I am getting the same problem in epoch 2<br>
RuntimeError: DataLoader worker (pid 26810) is killed by signal: Killed.<br>
Anyone has suggestions ?</p>