Training time gets slower and slower on CPU

ocanevet: <p>Hi all,</p>
<p>I am facing an issue when training an autoencoder on CPU (I am designing a lab for students to be made on a platform with no GPU, but the problem I will describe does not happen on GPU).</p>
<p>After some point, the time of an epoch starts increasing a lot. I provide a minimal working example code to reproduce the issue. At the beginning, the loop “get the batch, forward, grad, and optim step” takes 0.25 seconds, and after some time, it can be 2 or 3 times higher. This is what the graph below shows:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/6/c609a640ff78ec3baaa83a1e4bf1d26cbfee4183.png" data-download-href="https://discuss.pytorch.org/uploads/default/c609a640ff78ec3baaa83a1e4bf1d26cbfee4183" title="pytorch-time-epoch-increasing"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c609a640ff78ec3baaa83a1e4bf1d26cbfee4183_2_499x375.png" alt="pytorch-time-epoch-increasing" data-base62-sha1="sfVdcBkJTjvvgDyome6B0chw1Pl" width="499" height="375" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c609a640ff78ec3baaa83a1e4bf1d26cbfee4183_2_499x375.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/6/c609a640ff78ec3baaa83a1e4bf1d26cbfee4183_2_748x562.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/6/c609a640ff78ec3baaa83a1e4bf1d26cbfee4183.png 2x" data-dominant-color="F8F9F8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch-time-epoch-increasing</span><span class="informations">800×600 61 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I tried many things including:</p>
<ul>
<li>call <code>del batch, loss</code> and <code>gc.collect()</code> as suggested here <a href="https://discuss.pytorch.org/t/i-run-out-of-memory-after-a-certain-amount-of-batches-when-training-a-resnet18/1911/2" class="inline-onebox">I run out of memory after a certain amount of batches when training a resnet18 - #2 by smth</a></li>
<li>Remove variables which could keep computational graph in memory as suggested here  <a href="https://discuss.pytorch.org/t/training-gets-slow-down-by-each-batch-slowly/4460/21" class="inline-onebox">Training gets slow down by each batch slowly - #21 by yqwang</a> In particular, I removed statements like <code>total_loss += loss.item()</code> just in case that would not detach the loss from the graph, also removed <code>nn.MSELoss()</code>, the initial data-loader, and in the code below I used a random tensor as a minimal working example, but the problem started with a “real” data set.</li>
<li>The closest issue I found is <a href="https://discuss.pytorch.org/t/why-the-training-slow-down-with-time-if-training-continuously-and-gpu-utilization-begins-to-jitter-dramatically/11444/4" class="inline-onebox">Why the training slow down with time if training continuously? And Gpu utilization begins to jitter dramatically? - #4 by Musoy_King</a> but was related to CUDA.</li>
<li>I commented out the <code>optimizer.step()</code> and the issue disappeared, but obviously the model is not training anymore.</li>
<li>use powers of 2 in all parameters</li>
</ul>
<p>The issue arises on “PyTorch 1.10.2 + CUDA 11.3”, on “PyTorch 1.10.2 cpu only” and on “PyTorch with CUDA 11.1 on Google colab”.</p>
<p>Here is a minimal working example to reproduce the issue:</p>
<pre data-code-wrap="python"><code class="lang-python"># coding=utf-8

import argparse
import time
import gc
import torch
import torch.nn as nn

class AutoEncoder(nn.Module):
    def __init__(self, h=128, e=128):
        super().__init__()
        self.encoder = nn.Sequential(
            nn.Conv2d(1, h, 5, 1, 0), nn.ReLU(),
            nn.Conv2d(h, h, 5, 1, 0), nn.ReLU(),
            nn.Conv2d(h, h, 4, 2, 0), nn.ReLU(),
            nn.Conv2d(h, h, 3, 2, 0), nn.ReLU(),
            nn.Conv2d(h, e, 5, 1, 0),
        )
        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(e, h, 5, 1, 0), nn.ReLU(),
            nn.ConvTranspose2d(h, h, 3, 2, 0), nn.ReLU(),
            nn.ConvTranspose2d(h, h, 4, 2, 0), nn.ReLU(),
            nn.ConvTranspose2d(h, h, 5, 1, 0), nn.ReLU(),
            nn.ConvTranspose2d(h, 1, 5, 1, 0),
        )

    def forward(self, x):
        x = self.decoder(self.encoder(x))
        return x

model = AutoEncoder(32, 32)
print(model)
optimizer = torch.optim.Adam(model.parameters(), lr=0.002, weight_decay=1e-4)
train_images = torch.randn(16384, 1, 32, 32)
for epoch in range(40):
    for i, batch in enumerate(train_images.split(64)):
        tic = time.perf_counter()
        output = model(batch)
        loss = 0.5 * (output - batch).pow(2).sum() / batch.size(0)
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()
        toc = time.perf_counter()
        gc.collect()
        del loss, output, batch

        print(f"epoch {epoch} {i} took {toc-tic:.2f} seconds")
</code></pre>
<p>Again, I have cleaned the code to show the issue. Do you have any idea? As I am designing a lab for students, I would like to avoid hacks like “reload the model every epoch to keep the training fast”.</p>
<p>Thanks for your help.</p>

ptrblck: <p>Could you check if your workstation is reducing its clocks due to overheating etc.?<br>
Based on your description I would probably start by profiling the system and make sure it can run at a high load for an extended period of time.</p>

ocanevet: <p>Thanks a lot for your reply. I have tried to monitor the frequency of the CPU. I don’t know if I did that properly, I used <code>psutil.cpu_freq().current</code>.</p>
<p>However, I have ran the same experiment on many different platforms and the behaviour is consistent across them. The reported frequency does not seem to change (but I don’t know if this how we can check if a processor is down-clocking). But since the starting point of increasing time is always 1500, it would be strange if it did.</p>
<p>On the graph below,</p>
<ul>
<li>the x-axis is the number of minibatch used for the forward/backward/step,</li>
<li>the left y-axis is the time is take for one forward/backward/step</li>
<li>the right y-axis is the CPU frequency returned by <code>psutil.cpu_freq().current</code>.</li>
</ul>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/a/cafa5f26474cc3264bd91ba6b81a704d204ea3e9.png" data-download-href="https://discuss.pytorch.org/uploads/default/cafa5f26474cc3264bd91ba6b81a704d204ea3e9" title="plot"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/a/cafa5f26474cc3264bd91ba6b81a704d204ea3e9_2_666x500.png" alt="plot" data-base62-sha1="sXCRO7hdCtaGzWQXFEqzgnu5h9v" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/a/cafa5f26474cc3264bd91ba6b81a704d204ea3e9_2_666x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/a/cafa5f26474cc3264bd91ba6b81a704d204ea3e9_2_999x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/c/a/cafa5f26474cc3264bd91ba6b81a704d204ea3e9_2_1332x1000.png 2x" data-dominant-color="F5EFEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">plot</span><span class="informations">1600×1200 276 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>This is strangely very consistent across plateforms, when 1500 minibatches have been forwarded, the time increases. This is not the case on the AMD processor. On the JupyterHub/Kubernetes, the trend is less obvious, but still, the time increases a bit as of 1500.</p>
<p>I also tried (not shown here) PyTorch 1.1, 1.6, 1.7, 1.8, and 1.9 all showing the same trend.</p>
<p>Does it rind any bell to you? I am less sure about the AMD thing, as I am on a server and don’t master what is going on. I will try to find a laptop with an AMD proc to master what is running on the machine.</p>
<p>Thanks for your help.</p>

albanD: <p>Hi,</p>
<p>Could you try adding <code>torch.set_flush_denormal(True)</code> at the top of your script to see if that fixes the issue?</p>

ocanevet: <p>Thanks a lot for your suggestion: with <code>torch.set_flush_denormal(True)</code>, this has solved the issue. I ran each experiment multiple times and the results are consistent: no more increase in time after step 1500.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/4/d412c18d8e43f9239896eac65ed2b8195628b03f.png" data-download-href="https://discuss.pytorch.org/uploads/default/d412c18d8e43f9239896eac65ed2b8195628b03f" title="plot-with"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d412c18d8e43f9239896eac65ed2b8195628b03f_2_666x500.png" alt="plot-with" data-base62-sha1="ug5pjYkUwRjAmrDG99qSpHTAiUT" width="666" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d412c18d8e43f9239896eac65ed2b8195628b03f_2_666x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d412c18d8e43f9239896eac65ed2b8195628b03f_2_999x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/4/d412c18d8e43f9239896eac65ed2b8195628b03f_2_1332x1000.png 2x" data-dominant-color="EDEBEC"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">plot-with</span><span class="informations">1600×1200 296 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thanks a lot.</p>

albanD: <p>Great to hear!</p>
<p>For future times, this happens when numbers gets very close to 0. Your CPU, to try and keep precision, has to do some fancy arithmetic and thus gets slowed down.<br>
This happened to me before on “toy” examples where your model converges very well and thus a lot of numbers are very close to 0.</p>