Data Loader was killed

username: sunformoon
cooked: <p>If I set the num_workers to be 8 (or 4), then after hundreds of iterations, I got error “RuntimeError: DataLoader worker (pid 56901) is killed by signal: Killed.”</p>
<p>I searched related topics, some suggested “num_workers=0”. So I set it to be 0 but after hundreds of iterations, I still got killed. This time, only “Killed” is given. No other hints.</p>
<p>The top output and the training output are as follows:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/d6aca309a3b85c07ce07d6feb20284a82fae059a.png" data-download-href="https://discuss.pytorch.org/uploads/default/d6aca309a3b85c07ce07d6feb20284a82fae059a" title="18"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d6aca309a3b85c07ce07d6feb20284a82fae059a_2_690x99.png" alt="18" data-base62-sha1="uD63bNf5t00M9BtmHOZ2PXbEbLY" width="690" height="99" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d6aca309a3b85c07ce07d6feb20284a82fae059a_2_690x99.png, https://discuss.pytorch.org/uploads/default/optimized/2X/d/d6aca309a3b85c07ce07d6feb20284a82fae059a_2_1035x148.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/d/d6aca309a3b85c07ce07d6feb20284a82fae059a.png 2x" data-dominant-color="163814"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">18</span><span class="informations">1120×162 46.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/2/232dd1a935c1b5eacd1f639d0be1ac00b24f0d96.png" data-download-href="https://discuss.pytorch.org/uploads/default/232dd1a935c1b5eacd1f639d0be1ac00b24f0d96" title="03"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/2/232dd1a935c1b5eacd1f639d0be1ac00b24f0d96_2_518x500.png" alt="03" data-base62-sha1="51cSrMuWFedgobiltgUS5UKxfhA" width="518" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/2/232dd1a935c1b5eacd1f639d0be1ac00b24f0d96_2_518x500.png, https://discuss.pytorch.org/uploads/default/optimized/2X/2/232dd1a935c1b5eacd1f639d0be1ac00b24f0d96_2_777x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/2/232dd1a935c1b5eacd1f639d0be1ac00b24f0d96_2_1036x1000.png 2x" data-dominant-color="10220E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">03</span><span class="informations">1092×1054 235 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: SimonW
cooked: <p>Your data loading code is killed by SIGKILL, which is a very fatal signal. You should check and see how it can trigger such a signal.</p>

username: sunformoon
cooked: <p>Solved. Seems I have a bunch of tensors which need requires_grad to be False while I set it to be True.</p>
<p>For a reference, one update of the previous error I got: “RuntimeError: Torch: not enough memory: you tried to allocate 1GB. Buy new RAM!”</p>