Training runs infinitely without new epochs created

username: Fanny
cooked: <p>Hello everyone,<br>
I try to train a MaskRCNN ResNet50 model with ~5000 images, using torchvision (based on this tutorial [<a href="https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">TorchVision Object Detection Finetuning Tutorial — PyTorch Tutorials 2.2.0+cu121 documentation</a>]). After one day with no return, I get this:</p>
<pre><code class="lang-auto">Epoch: [0]  [   0/1643]  eta: 1:02:35  lr: 0.000010  loss: 3.7346 (3.7346)  loss_classifier: 1.6762 (1.6762)  loss_box_reg: 0.1342 (0.1342)  loss_mask: 1.7567 (1.7567)  loss_objectness: 0.1513 (0.1513)  loss_rpn_box_reg: 0.0163 (0.0163)  time: 2.2856  data: 0.1504  max mem: 6379
Epoch: [0]  [ 200/1643]  eta: 0:52:33  lr: 0.001009  loss: 0.8649 (1.2449)  loss_classifier: 0.1609 (0.2664)  loss_box_reg: 0.1137 (0.1101)  loss_mask: 0.5193 (0.7439)  loss_objectness: 0.0410 (0.1019)  loss_rpn_box_reg: 0.0137 (0.0225)  time: 2.3259  data: 0.0890  max mem: 10659
Epoch: [0]  [ 400/1643]  eta: 0:47:08  lr: 0.002008  loss: 0.8563 (1.0653)  loss_classifier: 0.1598 (0.2176)  loss_box_reg: 0.1226 (0.1216)  loss_mask: 0.5100 (0.6313)  loss_objectness: 0.0387 (0.0746)  loss_rpn_box_reg: 0.0077 (0.0201)  time: 2.5439  data: 0.0874  max mem: 10659
Epoch: [0]  [ 600/1643]  eta: 1 day, 4:12:41  lr: 0.003007  loss: 0.7584 (0.9834)  loss_classifier: 0.1494 (0.1987)  loss_box_reg: 0.1150 (0.1239)  loss_mask: 0.4441 (0.5806)  loss_objectness: 0.0200 (0.0611)  loss_rpn_box_reg: 0.0075 (0.0191)  time: 443.5763  data: 0.1023  max mem: 10659
</code></pre>
<p>And then nothing happens (I tried waiting for several days), there is no error and no more training steps calculated.</p>
<p>When I type nvidia-smi:</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------+
| NVIDIA-SMI 450.156.00   Driver Version: 450.156.00   CUDA Version: 11.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GRID T4-16C         On   | 00000000:02:01.0 Off |                    0 |
| N/A   N/A    P0    N/A /  N/A |  16274MiB / 16384MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      2324      C   ...v/envs/default/bin/python    14054MiB |
+-----------------------------------------------------------------------------+
</code></pre>
<p>I tried reducing batch size or number of images but I get the same problem each time.</p>
<p>I use torch 1.10.1 and torchvision 0.11.2 and I get this from torch:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c6c97bb188708e9eaee9167f957672752c11a11a.jpeg" alt="Capture2" data-base62-sha1="smydjWDMX32KJRe45hhNDUeVtq2" width="285" height="143"></p>
<p>Any idea what causes the problem ?<br>
Thanks</p>

username: ptrblck
cooked: <p>Could you just use a single image in the <code>Dataset</code> (keep the <code>Dataset</code>, <code>DataLoader</code> etc.) and see if the code is still stuck at the end of the epoch?<br>
If so, could you check if you are using any callbacks etc. at the end of an epoch and if this could cause the hang?</p>

username: Fanny
cooked: <p>I quickly got these logs:</p>
<pre><code class="lang-auto">/opt/pythenv/envs/default/lib64/python3.6/site-packages/torch/functional.py:445: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at  ../aten/src/ATen/native/TensorShape.cpp:2157.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
Epoch: [0]  [0/1]  eta: 0:00:00  lr: 0.005000  loss: 4.6190 (4.6190)  loss_classifier: 1.8476 (1.8476)  loss_box_reg: 0.0121 (0.0121)  loss_mask: 2.6329 (2.6329)  loss_objectness: 0.1016 (0.1016)  loss_rpn_box_reg: 0.0247 (0.0247)  time: 0.6683  data: 0.0355  max mem: 1506
Epoch: [0] Total time: 0:00:00 (0.6757 s / it)
creating index...
index created!
</code></pre>
<p>Then nothing happens, but I think it’s normal because after that it’s the evaluation part <code>engine.evaluate()</code> and I have nothing in my <code>data_loader_val</code>. I’m gonna check by adding one image in data_loader_val.</p>

username: Fanny
cooked: <p>So it works perfectly fine with one image. The training started immediately and finished in a minute.<br>
Could it be a problem with the images/annotations I give as input ? How could I check that ?</p>

username: ptrblck
cooked: <p>I don’t know what might be causing the issue, but based on your explanation it seems the only difference between the successful run and the hang is the size of the datasets?<br>
If so, are you able to iterate both <code>DataLoader</code>s (without training) or is this also hanging?</p>

username: Fanny
cooked: <p>So I think I figured out what was the issue. The NIVIDA license I used was expired and after renewing it, the training works again. However I don’t know how I was able to train a model with few images…</p>

username: ptrblck
cooked: <p>That’s interesting. What kind of “NVIDIA license” are you referring to?</p>