U-Net training (with Pytorch) taking up more time after ~100 iterations

Ella: <p>I am training a pretty standard  UNet model. The only extension is the loss function that is a composition of three other losses. Each iteration takes something like ~1.3 minutes, and I notice that after ~ 100 iterations,  the time for each iteration grows linearly. Any idea why it is happening?</p>
<p>PS. I read in forums that it can be due to lack of .item() in loss summation, i.e., total += loss.item()<br>
I added that .item(), but it doesn’t help.</p>
<p>Epoch [101]  Loss 0.0020: 100%|██████████| 1/1 [01:28&lt;00:00, 88.95s/it]<br>
Epoch [102]  Loss 0.0021: 100%|██████████| 1/1 [01:29&lt;00:00, 89.56s/it]<br>
Epoch [103]  Loss 0.0020: 100%|██████████| 1/1 [01:29&lt;00:00, 89.60s/it]<br>
Epoch [104]  Loss 0.0020: 100%|██████████| 1/1 [01:29&lt;00:00, 89.59s/it]<br>
Epoch [105]  Loss 0.0019: 100%|██████████| 1/1 [01:30&lt;00:00, 90.49s/it]<br>
Epoch [106]  Loss 0.0019: 100%|██████████| 1/1 [01:27&lt;00:00, 87.88s/it]<br>
Epoch [107]  Loss 0.0018: 100%|██████████| 1/1 [01:28&lt;00:00, 88.62s/it]<br>
Epoch [108]  Loss 0.0018: 100%|██████████| 1/1 [01:29&lt;00:00, 89.18s/it]<br>
Epoch [109]  Loss 0.0018: 100%|██████████| 1/1 [01:27&lt;00:00, 87.38s/it]<br>
Epoch [110]  Loss 0.0018: 100%|██████████| 1/1 [01:30&lt;00:00, 90.76s/it]<br>
Epoch [111]  Loss 0.0017: 100%|██████████| 1/1 [01:27&lt;00:00, 87.06s/it]<br>
Epoch [112]  Loss 0.0017: 100%|██████████| 1/1 [01:30&lt;00:00, 90.43s/it]<br>
Epoch [113]  Loss 0.0017: 100%|██████████| 1/1 [01:29&lt;00:00, 89.99s/it]<br>
Epoch [114]  Loss 0.0016: 100%|██████████| 1/1 [01:29&lt;00:00, 89.87s/it]<br>
Epoch [115]  Loss 0.0016: 100%|██████████| 1/1 [01:26&lt;00:00, 86.47s/it]<br>
Epoch [116]  Loss 0.0016: 100%|██████████| 1/1 [01:28&lt;00:00, 88.58s/it]<br>
Epoch [117]  Loss 0.0015: 100%|██████████| 1/1 [01:26&lt;00:00, 86.67s/it]<br>
Epoch [118]  Loss 0.0015: 100%|██████████| 1/1 [01:27&lt;00:00, 87.48s/it]<br>
Epoch [119]  Loss 0.0015: 100%|██████████| 1/1 [01:29&lt;00:00, 89.70s/it]<br>
Epoch [120]  Loss 0.0014: 100%|██████████| 1/1 [01:28&lt;00:00, 88.29s/it]<br>
Epoch [121]  Loss 0.0014: 100%|██████████| 1/1 [01:29&lt;00:00, 89.31s/it]<br>
Epoch [122]  Loss 0.0014: 100%|██████████| 1/1 [01:30&lt;00:00, 90.73s/it]<br>
Epoch [123]  Loss 0.0014: 100%|██████████| 1/1 [01:59&lt;00:00, 119.80s/it]<br>
Epoch [124]  Loss 0.0013: 100%|██████████| 1/1 [02:28&lt;00:00, 148.24s/it]<br>
Epoch [125]  Loss 0.0013: 100%|██████████| 1/1 [02:06&lt;00:00, 126.39s/it]<br>
Epoch [126]  Loss 0.0013: 100%|██████████| 1/1 [01:45&lt;00:00, 105.44s/it]<br>
Epoch [127]  Loss 0.0012: 100%|██████████| 1/1 [02:21&lt;00:00, 141.05s/it]<br>
Epoch [128]  Loss 0.0012: 100%|██████████| 1/1 [01:40&lt;00:00, 100.25s/it]<br>
Epoch [129]  Loss 0.0012: 100%|██████████| 1/1 [02:01&lt;00:00, 121.82s/it]<br>
Epoch [130]  Loss 0.0012: 100%|██████████| 1/1 [01:56&lt;00:00, 116.14s/it]<br>
Epoch [131]  Loss 0.0011: 100%|██████████| 1/1 [03:12&lt;00:00, 192.90s/it]<br>
Epoch [132]  Loss 0.0011: 100%|██████████| 1/1 [01:46&lt;00:00, 106.17s/it]<br>
Epoch [133]  Loss 0.0011: 100%|██████████| 1/1 [02:12&lt;00:00, 132.23s/it]<br>
Epoch [134]  Loss 0.0011: 100%|██████████| 1/1 [02:17&lt;00:00, 137.05s/it]<br>
Epoch [135]  Loss 0.0011: 100%|██████████| 1/1 [02:05&lt;00:00, 125.36s/it]<br>
Epoch [136]  Loss 0.0010: 100%|██████████| 1/1 [02:17&lt;00:00, 137.81s/it]<br>
Epoch [137]  Loss 0.0010: 100%|██████████| 1/1 [02:27&lt;00:00, 147.47s/it]<br>
Epoch [138]  Loss 0.0010: 100%|██████████| 1/1 [02:13&lt;00:00, 133.29s/it]<br>
Epoch [139]  Loss 0.0010: 100%|██████████| 1/1 [02:00&lt;00:00, 120.73s/it]<br>
Epoch [140]  Loss 0.0010: 100%|██████████| 1/1 [02:01&lt;00:00, 121.87s/it]<br>
Epoch [141]  Loss 0.0010: 100%|██████████| 1/1 [02:01&lt;00:00, 121.29s/it]<br>
Epoch [142]  Loss 0.0009: 100%|██████████| 1/1 [01:59&lt;00:00, 119.16s/it]<br>
Epoch [143]  Loss 0.0009: 100%|██████████| 1/1 [02:13&lt;00:00, 133.37s/it]<br>
Epoch [144]  Loss 0.0009: 100%|██████████| 1/1 [02:08&lt;00:00, 128.09s/it]<br>
Epoch [145]  Loss 0.0009: 100%|██████████| 1/1 [02:20&lt;00:00, 140.43s/it]<br>
Epoch [146]  Loss 0.0009: 100%|██████████| 1/1 [02:22&lt;00:00, 142.48s/it]<br>
Epoch [147]  Loss 0.0008: 100%|██████████| 1/1 [02:18&lt;00:00, 138.08s/it]<br>
Epoch [148]  Loss 0.0008: 100%|██████████| 1/1 [02:19&lt;00:00, 139.63s/it]<br>
Epoch [149]  Loss 0.0008: 100%|██████████| 1/1 [02:24&lt;00:00, 144.33s/it]<br>
Epoch [150]  Loss 0.0008: 100%|██████████| 1/1 [02:58&lt;00:00, 178.68s/it]<br>
Epoch [151]  Loss 0.0008: 100%|██████████| 1/1 [02:33&lt;00:00, 153.68s/it]<br>
Epoch [152]  Loss 0.0008: 100%|██████████| 1/1 [02:28&lt;00:00, 148.57s/it]<br>
Epoch [153]  Loss 0.0009: 100%|██████████| 1/1 [02:33&lt;00:00, 153.27s/it]<br>
Epoch [154]  Loss 0.0008: 100%|██████████| 1/1 [02:36&lt;00:00, 156.50s/it]<br>
Epoch [155]  Loss 0.0007: 100%|██████████| 1/1 [03:12&lt;00:00, 192.43s/it]<br>
Epoch [156]  Loss 0.0007: 100%|██████████| 1/1 [03:36&lt;00:00, 216.21s/it]<br>
Epoch [157]  Loss 0.0007: 100%|██████████| 1/1 [04:17&lt;00:00, 257.19s/it]<br>
Epoch [158]  Loss 0.0007: 100%|██████████| 1/1 [03:35&lt;00:00, 215.72s/it]<br>
Epoch [159]  Loss 0.0007: 100%|██████████| 1/1 [05:49&lt;00:00, 349.24s/it]<br>
Epoch [160]  Loss 0.0007: 100%|██████████| 1/1 [05:20&lt;00:00, 320.25s/it]<br>
Epoch [161]  Loss 0.0007: 100%|██████████| 1/1 [05:10&lt;00:00, 310.60s/it]<br>
Epoch [162]  Loss 0.0007: 100%|██████████| 1/1 [05:15&lt;00:00, 315.77s/it]<br>
Epoch [163]  Loss 0.0006: 100%|██████████| 1/1 [05:12&lt;00:00, 312.63s/it]<br>
Epoch [164]  Loss 0.0006: 100%|██████████| 1/1 [05:32&lt;00:00, 332.89s/it]<br>
Epoch [165]  Loss 0.0006: 100%|██████████| 1/1 [05:38&lt;00:00, 338.47s/it]<br>
Epoch [166]  Loss 0.0006: 100%|██████████| 1/1 [06:06&lt;00:00, 366.14s/it]<br>
Epoch [167]  Loss 0.0006: 100%|██████████| 1/1 [04:36&lt;00:00, 276.96s/it]<br>
Epoch [168]  Loss 0.0008: 100%|██████████| 1/1 [04:58&lt;00:00, 298.22s/it]<br>
Epoch [169]  Loss 0.0012: 100%|██████████| 1/1 [05:06&lt;00:00, 306.25s/it]<br>
Epoch [170]  Loss 0.0007: 100%|██████████| 1/1 [05:46&lt;00:00, 346.06s/it]<br>
Epoch [171]  Loss 0.0008: 100%|██████████| 1/1 [04:49&lt;00:00, 289.96s/it]<br>
Epoch [172]  Loss 0.0007: 100%|██████████| 1/1 [05:10&lt;00:00, 310.00s/it]<br>
Epoch [173]  Loss 0.0008: 100%|██████████| 1/1 [05:52&lt;00:00, 352.54s/it]</p>

ptrblck: <p>Make sure to avoid storing any tensors which are attached to the computation graph.<br>
I.e. if you are appending any tensor to e.g. a <code>list</code> or are accumulating them via e.g. <code>loss += batch_loss</code>, check if the used tensor has a valid <code>.grad_fn</code> and if so <code>.detach()</code> this tensor before storing it (assuming this fits your use case).<br>
If that’s not the case, then check if your system might be throttling due to e.g. overheating.</p>