The training speed suddenly become slow sometimes

Smith_Jack: <p>I write a small net and train it on a very simple problem, but sometimes I find that the speed of pytorch becomes very slow.<br>
I say SOMETIMES, because I can’t find the pattern of this behavior. I changed some parameters, for example, leaning rate or the number of layers of the net, this issue disappeared. And I changed those parameters back to their old settings, the slow-down problem didn’t appear again. So I don’t know what’ the problem.<br>
If I press Ctrl+C when the program get stuck, I find it <strong>usually</strong> stops at this line</p>
<pre data-code-wrap="bash"><code class="lang-bash">return F.linear(input, self.weight, self.bias)
</code></pre>
<p>Below is what I get If I press Ctrl+C</p>
<pre data-code-wrap="bash"><code class="lang-bash">Traceback (most recent call last):
  File "/mnt/e/workdir/ML_ADS/gauss/diff_min/diff_sin.py", line 134, in &lt;module&gt;
    x0_2 = model(x, x+noise, q)
           ^^^^^^^^^^^^^^^^^^^^
  File "/mnt/d/software_install/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/d/software_install/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/e/workdir/ML_ADS/gauss/diff_min/diff_sin.py", line 115, in forward
    x0_2 = self.dirformer(z, x)
           ^^^^^^^^^^^^^^^^^^^^
  File "/mnt/d/software_install/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/d/software_install/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/e/workdir/ML_ADS/gauss/diff_min/diff_sin.py", line 106, in forward
    return self.mlp(zx)
           ^^^^^^^^^^^^
  File "/mnt/d/software_install/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/d/software_install/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/d/software_install/pytorch/lib/python3.12/site-packages/torch/nn/modules/container.py", line 217, in forward
    input = module(input)
            ^^^^^^^^^^^^^
  File "/mnt/d/software_install/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/d/software_install/pytorch/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/d/software_install/pytorch/lib/python3.12/site-packages/torch/nn/modules/linear.py", line 116, in forward
    return F.linear(input, self.weight, self.bias)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt

</code></pre>

ptrblck: <p>To narrow down the randomly occurring slowdown, you could profile your workload and check which part of your training code is causing the drop in speed.</p>