"Suggested max num workers is 2" but I have 96 cores?

drscotthawley: <p>Hi, I’m confused. I’ve got 96 CPUs with 2 threads each.<br>
I chose num_workers=12 for my DataLoader but I’m getting a warning:</p>
<blockquote>
<p>UserWarning: This DataLoader will create 12 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.<br>
warnings.warn(_create_warning_msg(</p>
</blockquote>
<p>Why is the suggested number only 2?</p>
<p>I can see from the <a href="https://github.com/pytorch/pytorch/blob/9cf84347767c8abb8feba18a9a1baba321eeb8b9/torch/utils/data/dataloader.py#L543" rel="noopener nofollow ugc"><code>DataLoader</code> source code</a>  that it sets</p>
<pre data-code-wrap="python"><code class="lang-python">max_num_worker_suggest = len(os.sched_getaffinity(0))
</code></pre>
<p>and when I run <code>len(os.sched_getaffinity(0))</code> manually I get 96.</p>

drscotthawley: <p>On closer inspection, this seems to be a <code>jupyter notebook</code> problem.</p>
<p>I get 96 when I run <code>len(os.sched_getaffinity(0))</code> in the Python REPL from the command line.</p>
<p>The same command executed in a jupyter notebook (in the same python environment) returns only 2.     ??</p>
<p>(Am now looking around trying to figure out how to keep jupyter from limiting my CPUs to 2, and not finding much encouraging news on StackExchange so far.)</p>

drscotthawley: <p>running</p>
<pre><code class="lang-python">os.system("taskset -c -p 0-95 %d" % os.getpid())
</code></pre>
<p>from within jupyter got the number up to 96.</p>

pvtien96: <p>I’m encountering a similar problem on a HPC with 80 cores:</p>
<pre data-code-wrap="bash"><code class="lang-bash">/ccc/products2/python3-3.10.6/Rhel_8__x86_64/gcc--8.3.0__openmpi--4.0.1/cuda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:560: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(
/ccc/products2/python3-3.10.6/Rhel_8__x86_64/gcc--8.3.0__openmpi--4.0.1/cuda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:560: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(
/ccc/products2/python3-3.10.6/Rhel_8__x86_64/gcc--8.3.0__openmpi--4.0.1/cuda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:560: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(
/ccc/products2/python3-3.10.6/Rhel_8__x86_64/gcc--8.3.0__openmpi--4.0.1/cuda/lib/python3.10/site-packages/torch/utils/data/dataloader.py:560: UserWarning: This DataLoader will create 16 worker processes in total. Our suggested max number of worker in current system is 2, which is smaller than what this DataLoader is going to create. Please be aware that excessive worker creation might get DataLoader running slow or even freeze, lower the worker number to avoid potential slowness/freeze if necessary.
  warnings.warn(_create_warning_msg(
</code></pre>
<p>I’m using the default <a href="https://github.com/pytorch/vision/blob/main/references/classification/train.py" rel="noopener nofollow ugc">training script</a> of torchvision.</p>

BramVanroy: <p>Did you ever figure this out? I am encountering the same issue in an HPC environment. When I ssh into a node and execute len(os.sched_getaffinity(0)) it correctly shows <code>48</code> but even with num_workers=4 I still get that the recommended num_workers=1.</p>

AnsonSavage: <p>I’m having a similar issue and I’m curious… <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=14" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>