My system is Windows, and when num_work is greater than 0 (my tests are 4 and 8), I will always call the py file of the training code

yinjun1131: <p>This is an interesting issue that I discovered when using yolov10. When my num_work is configured to 0, the training code executes normally. However, when the num_work parameter is 4/8, when my code reaches init under InfiniteDataLoader, self. iterator=super() is called__ When using the item__ () method, I immediately create a process with a number of num_works to call the py file I used to train the code. This repeated process ultimately leads to system OOM. I really want to record a video of this interesting issue, but unfortunately, I don’t know how to post it. I have to take screenshots of the important part of the log code and send them out</p>
<p>I have heard before that it is best not to set num_work to a value greater than 0 in Windows. Now, this question makes me curious. Why can’t this parameter be set to greater than 0 in Windows? From a bottom-up design perspective</p>
<p>this is when num_work was zeros:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/d/7d663981727ee03959a6a41963cfde21f01a2c59.png" data-download-href="https://discuss.pytorch.org/uploads/default/7d663981727ee03959a6a41963cfde21f01a2c59" title="1719241711128"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7d663981727ee03959a6a41963cfde21f01a2c59_2_690x273.png" alt="1719241711128" data-base62-sha1="hTkHldHARcg69ObHwakKU5H3Ljb" width="690" height="273" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7d663981727ee03959a6a41963cfde21f01a2c59_2_690x273.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/d/7d663981727ee03959a6a41963cfde21f01a2c59_2_1035x409.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/d/7d663981727ee03959a6a41963cfde21f01a2c59.png 2x" data-dominant-color="333538"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1719241711128</span><span class="informations">1273×505 112 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>this is when num_work was 8 or 4</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b45409dfff5300a3ae205a2900a62177ebcb30e.png" data-download-href="https://discuss.pytorch.org/uploads/default/7b45409dfff5300a3ae205a2900a62177ebcb30e" title="1719241884098"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b45409dfff5300a3ae205a2900a62177ebcb30e_2_667x500.png" alt="1719241884098" data-base62-sha1="hAv6i3rzVqGcjVmLC4HG5BPgzOm" width="667" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b45409dfff5300a3ae205a2900a62177ebcb30e_2_667x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/b/7b45409dfff5300a3ae205a2900a62177ebcb30e_2_1000x750.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b45409dfff5300a3ae205a2900a62177ebcb30e.png 2x" data-dominant-color="464647"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">1719241884098</span><span class="informations">1281×960 170 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <p>From the <a href="https://pytorch.org/docs/stable/notes/windows.html#multiprocessing-error-without-if-clause-protection">Windows FAQ</a>:</p>
<blockquote>
<p>The implementation of <code>multiprocessing</code> is different on Windows, which uses <code>spawn</code> instead of <code>fork</code>. So we have to wrap the code with an if-clause to protect the code from executing multiple times.</p>
</blockquote>
<p>Add the <code>if-clause</code> protection to avoid this behavior.</p>

yinjun1131: <p>But I am still very interested in this bug, why does it appear on Windows repeatedly calling the main process? Don’t you know how to do it on Ubuntu?</p>

ptrblck: <p>The quoted docs explain that the <code>spawn</code> method is used by default in Windows and you can try to set it explicitly on Linux to recreate the mentioned issues.</p>