Num_workers questions

Ardeal: <p>I tried the following:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th>items</th>
<th>num_workers=1</th>
<th>num_workers = 2</th>
<th>num_workers = 4</th>
<th>num_workers = 8</th>
</tr>
</thead>
<tbody>
<tr>
<td>CPU</td>
<td>10700K</td>
<td>10700K</td>
<td>10700K</td>
<td>10700K</td>
</tr>
<tr>
<td>CPU   RAM(G)</td>
<td>16</td>
<td>16</td>
<td>48</td>
<td>48</td>
</tr>
<tr>
<td>CPU RAM   used(G)</td>
<td>9.5</td>
<td>9.5</td>
<td>12</td>
<td>CUDA out of memory issue</td>
</tr>
<tr>
<td>GPU</td>
<td>3090</td>
<td>3090</td>
<td>3090</td>
<td>3090</td>
</tr>
<tr>
<td>GPU   RAM(G)</td>
<td>24</td>
<td>24</td>
<td>24</td>
<td>24</td>
</tr>
<tr>
<td>GPU RAM   used(G)</td>
<td>17.5</td>
<td>17.5</td>
<td>17.9</td>
<td>CUDA out of memory issue</td>
</tr>
<tr>
<td>GPU   Power(w)</td>
<td>257</td>
<td>357</td>
<td>357</td>
<td>CUDA out of memory issue</td>
</tr>
<tr>
<td>image count</td>
<td>10180</td>
<td>10180</td>
<td>10180</td>
<td>10180</td>
</tr>
<tr>
<td>epoch</td>
<td>100</td>
<td>100</td>
<td>100</td>
<td>100</td>
</tr>
<tr>
<td>batch   size</td>
<td>32</td>
<td>32</td>
<td>32</td>
<td>32</td>
</tr>
<tr>
<td>batch   count</td>
<td>319</td>
<td>319</td>
<td>319</td>
<td>319</td>
</tr>
<tr>
<td>total   time for training(hours)</td>
<td>15</td>
<td>6.459</td>
<td>6.05</td>
<td></td>
</tr>
<tr>
<td>time for   each epoch(minutes)</td>
<td>9</td>
<td>3.875</td>
<td>3.6333</td>
<td></td>
</tr>
<tr>
<td>average   time for batch(second)</td>
<td>1.69</td>
<td>0.73</td>
<td>0.617</td>
<td></td>
</tr>
</tbody>
</table>
</div><p>I have a few questions:<br>
<code></code></p><code>
<ol>
<li>the training speed of num_workers=2 is quite faster than that of num_workers=1, however, there is small difference for num_workers=2 and 4.</li>
<li>when num_workers=4, there are much CPU ram and GPU ram left, num_workers=8 doesn’t work. it is very weird.</li>
<li>num_workers assigns a few threads to load image to RAM(CPU). If batch size keeps 32, the RAM needed for CPU should not change. however, more RAM is used for num_workers = 4.</li>
</ol>
</code>
<p>The following is the issue for num_workers=8:</p>
<pre><code class="lang-python">Analyzing anchors... anchors/target = 5.86, Best Possible Recall (BPR) = 1.0000
Image sizes 640 train, 640 test
Using 4 dataloader workers
Logging results to runs\train\exp7
Starting training for 100 epochs...
     Epoch   gpu_mem       box       obj       cls     total   targets  img_size
  0%|          | 0/319 [00:00&lt;?, ?it/s]Note: NumExpr detected 16 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
NumExpr defaulting to 8 threads.
  0%|          | 0/319 [00:02&lt;?, ?it/s]
Traceback (most recent call last):
  File "D:/code_python/har_hailiang/har_hdd/algo/train.py", line 288, in train
    pred = model(imgs)  # forward
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "D:\code_python\har_hailiang\har_hdd\algo\models\yolo.py", line 122, in forward
    return self.forward_once(x, profile)  # single-scale inference, train
  File "D:\code_python\har_hailiang\har_hdd\algo\models\yolo.py", line 138, in forward_once
    x = m(x)  # run
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "D:\code_python\har_hailiang\har_hdd\algo\models\common.py", line 35, in forward
    return self.act(self.bn(self.conv(x)))
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\module.py", line 727, in _call_impl
    result = self.forward(*input, **kwargs)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\conv.py", line 423, in forward
    return self._conv_forward(input, self.weight)
  File "C:\ProgramData\Anaconda3\lib\site-packages\torch\nn\modules\conv.py", line 419, in _conv_forward
    return F.conv2d(input, weight, self.bias, self.stride,
RuntimeError: CUDA out of memory. Tried to allocate 3.85 GiB (GPU 0; 24.00 GiB total capacity; 1.47 GiB already allocated; 20.35 GiB free; 1.54 GiB reserved in total by PyTorch)
python-BaseException
</code></pre>

surya00060: <p>Why is there CUDA out of memory issue when you increase the num_workers? There shouldn’t be OOM error when num_workers is increased as it’s not related to GPU at all.</p>
<p>Can you check whether you are changing batch size or model input when increasing the num_workers.</p>

Ardeal: <p><a class="mention" href="/u/surya00060">@surya00060</a> ,<br>
Thank you!<br>
I have the same doubts as yours. however, I tried many times and encountered the same issue I pasted above.</p>
<p>I confirm: I have tried many times with only num_workers changed without any other code changed.</p>
<p>By the way, I was doing above training on Windows 10 but  not Linux.<br>
I estimate the above issue could be easily reproduced.</p>

Alexey_Demyanchuk: <p>Hi,<br>
May it be, you are doing some image transforms on gpu? As transforms are usually made by dataloader if you moving data to gpu somethere in your dataset code, increasing the number of workers will increase the number of dataloader replicas and so allocated gpu memory will be increased as well.</p>

Alexey_Demyanchuk: <aside class="quote no-group" data-username="Ardeal" data-post="1" data-topic="114130">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ardeal/48/33257_2.png" class="avatar"> Ardeal:</div>
<blockquote>
<p>the training speed of num_workers=2 is quite faster than that of num_workers=1, however, there is small difference for num_workers=2 and 4.</p>
</blockquote>
</aside>
<p>It is enough to have 2 workers to prepare the data so to fully utilize GPU.</p>
<aside class="quote no-group" data-username="Ardeal" data-post="1" data-topic="114130">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ardeal/48/33257_2.png" class="avatar"> Ardeal:</div>
<blockquote>
<p>when num_workers=4, there are much CPU ram and GPU ram left, num_workers=8 doesn’t work. it is very weird.</p>
</blockquote>
</aside>
<p>It is not clear from your data. Actually, the amount of used memory (both CPU and GPU) increases then you increase number of workers. Also, first 2 columns have CPU RAM 16 and second 2 - 48. Are those two different machines?</p>
<aside class="quote no-group" data-username="Ardeal" data-post="1" data-topic="114130">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ardeal/48/33257_2.png" class="avatar"> Ardeal:</div>
<blockquote>
<p>num_workers assigns a few threads to load image to RAM(CPU). If batch size keeps 32, the RAM needed for CPU should not change. however, more RAM is used for num_workers = 4.</p>
</blockquote>
</aside>
<p>Not exactly true. Workers prepare data in advance to push it into GPU as fast as possible so to utilize GPU the most. All those workers require memory to put processed data from disk to memory, so increasing number of workers will increase memory footprint.</p>

Ardeal: <p><a class="mention" href="/u/alexey_demyanchuk">@Alexey_Demyanchuk</a> ,<br>
Thank you!</p>
<p>For the first 2 columns, the RAM is 16G. when I changed num_worker=4, there is OOM issue. So, I add a new 32G RAM to my machine. So, those 4 columns are run in the same machine.</p>
<p>You explanation above is the only possibility that we could think of.</p>
<p>what is your machine configuration?<br>
what is your test data? for example, what is the time needed for each iteration/batch?</p>

Alexey_Demyanchuk: <aside class="quote no-group" data-username="Ardeal" data-post="6" data-topic="114130">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ardeal/48/33257_2.png" class="avatar"> Ardeal:</div>
<blockquote>
<p>what is your machine configuration?<br>
what is your test data? for example, what is the time needed for each iteration/batch?</p>
</blockquote>
</aside>
<p>I don’t work with this task. And always use just google colab. My explanation is just how I understand the topic from reading the documentation, other discussions and my own experience.</p>
<p>As for debugging your issue, I would suggest trying to check your data pipeline to understand why is it so memory hungry. It might be there is an opportunity to optimize it. Also, It seems like 2 workers are actually enough for the task, as you can see that overall training time doesn’t improve much with further increasing number of workers.</p>

Ardeal: <p><a class="mention" href="/u/alexey_demyanchuk">@Alexey_Demyanchuk</a> ,</p>
<p>Thank you!<br>
I am training yolov3 with input image size 640*640 to the network.</p>
<p>I agree with you.</p>
<aside class="quote no-group" data-username="Alexey_Demyanchuk" data-post="7" data-topic="114130">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alexey_demyanchuk/48/25254_2.png" class="avatar"> Alexey_Demyanchuk:</div>
<blockquote>
<p>Also, It seems like 2 workers are actually enough for the task, as you can see that overall training time doesn’t improve much with further increasing number of workers.</p>
</blockquote>
</aside>
<p>with 2 workers, the CPU and GPU might be fully used. however, I would like to know more details about  how GPU and CPU are used. I would also like to know where the bottleneck in hardware is.</p>
<p>for my case,<br>
during inference, if I run a video for inference(forward), the time needed is around 20ms.<br>
during training, the time needed for one batch is around 0.73s=730ms which includes the time for forward and backward. If the time for forward is 20ms, the time for backward is around 710ms. 20ms VS 710ms, this seems to be unreasonable for powerful 3090 GPU. This is my major concern about the data in my table.</p>
<p>I would like to know your data. I think the ratio of the time needed for forward and back on your machine should be similar as mine. Furthermore, the time for forward and backward should be comparable for yours and mine.</p>

Alexey_Demyanchuk: <p>I would like to elaborate a little on this. First, I don’t think it is as simple as 730 ms - 20 ms = 710 ms for backward pass, unless those 730 ms you measure only GPU part of the work. Second, you almost always have two phases of the work:</p>
<ol>
<li>You prepare data (reading, resizing, augmenting images goes here), this work usually done by CPU</li>
<li>You train a neural network (those are forward and backward pass), this work usually done by GPU</li>
</ol>
<p>To measure time spent throughout those task, you would probably better measure “data” time and “model” time separately. These separate measurements will help you to understand there the most of the work is taking place actually.</p>
<p>Consider my example. I am training a regular convnet classifier for kaggle competition with the backbone of ResNet200d. My setup is Google Colab V100 16 GB GPU. I use image size 768x768. Training takes ~17 images per second, validation ~ 37 images per second. I am actually not measuring the “data” part separately, so I can not tell you exact amount of time spent on GPU.</p>
<p>And you still can compare those numbers only then GPU utilization is the same, and you don’t have data processing as a bottleneck.</p>
<p>A valid solution on how to measure the time used by GPU will be to try it on synthetic data. Create tensors in memory and measure GPU time.</p>

Ardeal: <p><a class="mention" href="/u/alexey_demyanchuk">@Alexey_Demyanchuk</a> ，<br>
Thank you!</p>
<p>I did another interesting experiment on my another computer:<br>
CPU i9-9900K, ram 32G, GPU 2080Ti(11G),<br>
I trained the same yolov3 code, but with coco datasets.<br>
the batch size is 8, and there are totally 122218 images that are separated to 15278 batches.  num_workers =8, it could be successfully trained. Furthermore, the training time for each batch is around 0.28s.</p>
<code>
this experiment confirm my doubts:  
<ol>
<li>the training speed on my 3090 could be faster. there might be some hardware or software bottleneck.</li>
<li>my 3090 computer has much more hardware resource than my 2080Ti(11G) computer. however, on 3090 computer, num_workers could be set to 4 but not 8. there must be something that I should explore.</li>
</ol>
</code>

Alexey_Demyanchuk: <p>That would be more reliable to compare those two different machines on the same task.</p>

Ardeal: <p><a class="mention" href="/u/alexey_demyanchuk">@Alexey_Demyanchuk</a> ,<br>
Thank you for your reply!</p>
<p>I did the following test:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th style="text-align:center"></th>
<th style="text-align:center">num_workers=8</th>
<th style="text-align:center">num_workers = 8</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align:center">CPU</td>
<td style="text-align:center">i9-9900K(8 cores 16 threads)</td>
<td style="text-align:center">i7-10700K(8scores 16 threads)</td>
</tr>
<tr>
<td style="text-align:center">GPU</td>
<td style="text-align:center">2080Ti(11GB)</td>
<td style="text-align:center">3090(24GB)</td>
</tr>
<tr>
<td style="text-align:center">CPU RAM used</td>
<td style="text-align:center">9.5G</td>
<td style="text-align:center">unknown</td>
</tr>
<tr>
<td style="text-align:center">GPU RAM used</td>
<td style="text-align:center">17.5G</td>
<td style="text-align:center">unknown</td>
</tr>
<tr>
<td style="text-align:center">epoch</td>
<td style="text-align:center">300</td>
<td style="text-align:center">300</td>
</tr>
<tr>
<td style="text-align:center">datasets</td>
<td style="text-align:center">COCO</td>
<td style="text-align:center">COCO</td>
</tr>
<tr>
<td style="text-align:center">image count</td>
<td style="text-align:center">122218</td>
<td style="text-align:center">122218</td>
</tr>
<tr>
<td style="text-align:center">batch size</td>
<td style="text-align:center">8</td>
<td style="text-align:center">8</td>
</tr>
<tr>
<td style="text-align:center">code</td>
<td style="text-align:center">yolov3 the same</td>
<td style="text-align:center">yolov3 the same</td>
</tr>
</tbody>
</table>
</div><p>on 2080Ti computer, the code could be correctly trained.<br>
On 3090 computer, once the taining and validation data were loaded, the code doesn’t go forward. I click the “Pause Program” button on Pycharm, and found that the code stops in the following code:</p>
<pre><code class="lang-auto">C:\ProgramData\Anaconda3\Lib\multiprocessing\connection.py:
    def _exhaustive_wait(handles, timeout):
        # Return ALL handles which are currently signalled.  (Only
        # returning the first signalled might create starvation issues.)
        L = list(handles)
        ready = []
        while L:
            res = _winapi.WaitForMultipleObjects(L, False, timeout)
            if res == WAIT_TIMEOUT:
                break
            elif WAIT_OBJECT_0 &lt;= res &lt; WAIT_OBJECT_0 + len(L):
                res -= WAIT_OBJECT_0
            elif WAIT_ABANDONED_0 &lt;= res &lt; WAIT_ABANDONED_0 + len(L):
                res -= WAIT_ABANDONED_0
            else:
                raise RuntimeError('Should not get here')
            ready.append(L[res])
            L = L[res+1:]
            timeout = 0
        return ready
</code></pre>
<p>It seems that the code is waiting for thread?</p>

Alexey_Demyanchuk: <p>Hi,</p>
<p>I am a bit lost. Because in the beginning of the thread your code worked on 3090 machine.<br>
Hard to find the problem, sorry.</p>

Ardeal: <p><a class="mention" href="/u/alexey_demyanchuk">@Alexey_Demyanchuk</a> ,</p>
<p>In the beginning, I did work on 3090 machine. however, I couldn’t find out the resolution of the issue. I then tried it on 2080Ti machine.</p>
<p>After testing the code on 2080Ti, There is no issue on 2080Ti. the issue still exists on 3090 machine. It is the issue as the beginning.</p>

Alexey_Demyanchuk: <p>Is it the newest version of Pytorch you are using on the 3090 machine? If not, may be consider to update?</p>