Training "stucks" every three batches (pytorch 2.0)

username: jack_S
cooked: <p>I was training <code>resnet50</code> with <code>ImageNet</code> on <code>NVIDIA A40</code>. I found that my training speed slowed down every three batchs then recovered normal speed. Why would this happen?</p>
<p>Details:</p>
<ol>
<li>My dataloaders and models are like:<br>
<code>loader = DataLoader(dataset, batchsize, shuffle = True, num_workers = 4, prefetch_factor = 2, drop_last = True)</code><br>
<code>model = torch.compile(resnet50, mode = 'max-autotune')</code></li>
<li>I used <code>amp.autocast()</code> to accelerate my training.</li>
<li>The loss function is just <code>nn.CrossEntropyLoss</code>.</li>
<li>I set <code>cudnn.benchmark = True</code>.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/a/4a6cf71fbf0a8ec6676612b8748f8fc10a5069b3.png" data-download-href="https://discuss.pytorch.org/uploads/default/4a6cf71fbf0a8ec6676612b8748f8fc10a5069b3" title="pytorch"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4a6cf71fbf0a8ec6676612b8748f8fc10a5069b3_2_690x406.png" alt="pytorch" data-base62-sha1="aCoMYzHVCwxbgCYoe8yEZ7kEOav" width="690" height="406" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/a/4a6cf71fbf0a8ec6676612b8748f8fc10a5069b3_2_690x406.png, https://discuss.pytorch.org/uploads/default/original/3X/4/a/4a6cf71fbf0a8ec6676612b8748f8fc10a5069b3.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/a/4a6cf71fbf0a8ec6676612b8748f8fc10a5069b3.png 2x" data-dominant-color="FCFCFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">pytorch</span><span class="informations">873×514 21.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></li>
</ol>

username: AbdulsalamBande
cooked: <p>The DataLoader is set to use 4 workers with a prefetch factor of 2. It’s possible that the data loading process is not keeping up with the training process, causing the GPU to wait for new data every few batches. You can try the below points</p>
<ul>
<li>Increasing the <code>num_workers</code> to a higher value (e.g., 8 or 16) to see if it improves the situation.</li>
<li>Increasing the <code>prefetch_factor</code> so that more samples are preloaded.</li>
<li>Monitoring the CPU and disk usage to see if there’s a bottleneck in data loading.</li>
<li>You can try disabling <code>amp.autocast()</code> to see if the issue still persists. If the issue disappears, you might need to fine-tune the mixed precision training configuration.</li>
<li>You have set <code>cudnn.benchmark = True</code> , which allows CuDNN to automatically select the best algorithm for the current hardware configuration. In some cases, this can cause fluctuations in performance. You can try setting <code>cudnn.benchmark = False</code> and <code>cudnn.deterministic = True</code> to see if this improves the situation</li>
<li>Monitor the GPU utilization using a tool like <code>nvidia-smi</code> to see if there are any fluctuations in the GPU usage that correlate with the slowdowns.</li>
</ul>

username: jack_S
cooked: <p>Thanks! The training speed did increase after I tried some of these strategies. However, there’s another weird thing: the speed was good in the first several dozens of batches (about 50~100), but after these batches, the speed decreased, and the problem arose again. I have no idea about why this happened.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="AbdulsalamBande" data-post="2" data-topic="176759">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/abdulsalambande/48/40450_2.png" class="avatar"> AbdulsalamBande:</div>
<blockquote>
<p>You can try disabling <code>amp.autocast()</code> to see if the issue still persists. If the issue disappears, you might need to fine-tune the mixed precision training configuration.</p>
</blockquote>
</aside>
<p>Could you explain what “fine-tune the mixed precision training configuration” means here, please?</p>
<aside class="quote no-group" data-username="jack_S" data-post="4" data-topic="176759">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jack_s/48/59032_2.png" class="avatar"> jack_S:</div>
<blockquote>
<p>The training speed did increase after I tried some of these strategies</p>
</blockquote>
</aside>
<p>What exactly did you change?</p>

username: AbdulsalamBande
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="5" data-topic="176759">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Could you explain what “fine-tune the mixed precision training configuration” means here, please?</p>
</blockquote>
</aside>
<p>I meant to say trying lower-precision data types like float16 or higher-precision types like float32.</p>

username: jack_S
cooked: <p>I set <code>cudnn.deterministic = True</code> and increased <code>num_workers</code> then found the speed increased. However, the problem still exists now: the speed is good in first several dozens of batches but after these batches the speed decreases. It seems that about every "the power of 10"th (around 10th, 100th, 1000th, …) batch in one epoch the speed decreases by a half. And after the speed slowing down to some scale, the training gets “stuck” every three batches.</p>

username: AbdulsalamBande
cooked: <p>This might be a problem related to your data. I would suggest you inspect your data thoroughly. You can even try training with a smaller subset.</p>

username: ptrblck
cooked: <p>I think your data loading might run into a bottleneck and you could try to profile the <code>DataLoader</code> e.g. as used <a href="https://github.com/pytorch/examples/blob/54f4572509891883a947411fd7239237dd2a39c3/imagenet/main.py#L315-L318">here</a>.<br>
If you are seeing the slowdown in the <code>DataLoader</code> you could try to profile the data loading pipeline standalone to check what exactly is slowing it down.</p>

username: jack_S
cooked: <p>Thanks and you are right. The problem is in <code>Dataloader</code>s. Just 4 <code>num_worker</code>s and <code>prefetch_factor</code>s are not enough. My device has 12 workers and when I tried to use all of them the speed increased a lot. However there’s a new issue: if I use <code>model = torch.compile(model, mode = 'max-autotune')</code> with all 12 workers, then there will be a lot of errors like</p>
<pre data-code-wrap="Traceback"><code class="lang-plaintext">  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 328, in cudagraphify_impl
    static_outputs = model(list(static_inputs))
  File "/tmp/torchinductor_root/vi/cvigpzgiqkkwgcp62tygjtv2pdbmxoytpgcnbl52qjnqiq3m5u6u.py", line 2684, in call
    triton__16.run(buf39, buf37, primals_171, primals_172, buf42, buf40, buf41, buf43, 256, 1605632, grid=grid(256), stream=stream0)
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_inductor/triton_ops/autotune.py", line 190, in run
    result = launcher(
  File "&lt;string&gt;", line 6, in launcher
RuntimeError: Triton Error [CUDA]: operation failed due to a previous error during capture

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt/syc_cache/train.py", line 71, in &lt;module&gt;
    main()
  File "/mnt/syc_cache/train.py", line 20, in main
    train(args, p, epoch)
  File "/mnt/syc_cache/train.py", line 38, in train
    loss = forward(images, labels, args, p)
  File "/mnt/syc_cache/train.py", line 56, in forward
    output = p['model'](images)
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 82, in forward
    return self.dynamo_ctx(self._orig_mod.forward)(*args, **kwargs)
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 209, in _fn
    return fn(*args, **kwargs)
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torchvision/models/resnet.py", line 284, in forward
    def forward(self, x: Tensor) -&gt; Tensor:
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 209, in _fn
    return fn(*args, **kwargs)
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2819, in forward
    return compiled_fn(full_args)
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1222, in g
    return f(*args)
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2386, in debug_compiled_function
    return compiled_function(*args)
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1898, in runtime_wrapper
    all_outs = call_func_with_args(
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1247, in call_func_with_args
    out = normalize_as_list(f(args))
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1222, in g
    return f(*args)
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2151, in forward
    fw_outs = call_func_with_args(
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1247, in call_func_with_args
    out = normalize_as_list(f(args))
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 248, in run
    return model(new_inputs)
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 265, in run
    compiled_fn = cudagraphify_impl(model, new_inputs, static_input_idxs)
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 327, in cudagraphify_impl
    with torch.cuda.graph(graph, stream=stream):
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/cuda/graphs.py", line 173, in __exit__
    self.cuda_graph.capture_end()
  File "/root/miniconda3/envs/myconda/lib/python3.10/site-packages/torch/cuda/graphs.py", line 79, in capture_end
    super().capture_end()
RuntimeError: CUDA error: operation failed due to a previous error during capture
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.This text will be hidden
</code></pre>
<p>; if I just use <code>model = torch.compile(model)</code> with all 12 workers, then there will be no errors. Does <code>max-autotune</code> requires extra workers?</p>

username: ptrblck
cooked: <p>Does the code with with <code>torch.compile(model, mode="max-autotune")</code> if the <code>num_workers</code> are set to <code>0</code>?<br>
Based on the error message I would guess the failure is unrelated to the <code>DataLoader</code> workers and something fails inside <code>torch.compile</code>. However, the actual error is not displayed and only re-raised:</p>
<pre><code class="lang-python">RuntimeError: Triton Error [CUDA]: operation failed due to a previous error during capture
</code></pre>

username: jack_S
cooked: <p>The code can run with <code>torch.compile(model, mode="max-autotune")</code> with both <code>num_workers = 4, prefetch_factor = 4</code> and <code>num_workers = 0, prefetch_factor = None</code>. However, it fail with <code>num_workers = 8, prefetch_factor = 8</code> and higher.</p>
<p>The first <code>Traceback</code> in aforemetioned displayed errors only shows<br>
<code>RuntimeError: Triton Error [CUDA]: operation failed due to a previous error during capture</code>. The actual error is not displayed so I have no idea what causes this.</p>