Torch.compile + DDP: SIGSEGV/SIGTERM during inference step

username: sree_harsha
cooked: <p>Hi,</p>
<p>I am writing a training harness from scratch for work that involves iterative pruning – which uses DDP train each level.</p>
<p>tl;dr<br>
SIGTERM/SIGSEGV while running inference during a DDP run + model which has been torch.compile’d.</p>
<p>Exact error:</p>
<pre><code class="lang-auto">W0526 00:24:17.229000 22419848091456 torch/multiprocessing/spawn.py:145] Terminating processP ROCESS_ID via signal SIGTERM
Traceback (most recent call last):
  File "/harness.py", line 346, in &lt;module&gt;
    mp.spawn(main, args=(init_model, world_size, threshold, i), nprocs=world_size, join=True)
  File "/orch/multiprocessing/spawn.py", line 281, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
  File "torch/multiprocessing/spawn.py", line 237, in start_processes
    while not context.join():
  File "torch/multiprocessing/spawn.py", line 169, in join
    raise ProcessExitedException(
torch.multiprocessing.spawn.ProcessExitedException: process 1 terminated with signal SIGSEGV
</code></pre>
<p>Details:</p>
<p>To describe my setup:</p>
<ol>
<li>Model: ResNet-18/50 (with a mask per layers, same size as weight parameters as buffers).</li>
<li>Batch-Size: 256 (each)</li>
<li>Autocast: BFloat16</li>
<li>Dataset: ImageNet (224x224)</li>
<li>FFCV for dataloading (but for all purposes this shouldn’t matter)</li>
<li>torch.compile (mode = ‘reduce-overhead’) applied before passing the model to DDP as below:<br>
Tried with:</li>
<li>Different PyTorch versions: 2.0.1, 2.1, 2.2.1, 2.2.2 2.3</li>
<li>CUDA: 11.8 and 12.1</li>
</ol>
<pre><code class="lang-auto">self.model = torch.compile(self.model, mode='reduce-overhead')
self.model = DDP(self.model, device_ids=[self.gpu_id])
</code></pre>
<ol start="6">
<li>2x A100 GPUs (80 GB) with abundant memory/CPU resources.</li>
<li>num workers (if relevant): 8</li>
</ol>
<p>I launch training as follows:</p>
<pre><code class="lang-auto">for i, threshold in enumerate(thresholds):
        mp.spawn(main, args=(init_model, world_size, threshold, i), nprocs=world_size, join=True)
</code></pre>
<p>I can make no sense of this issue:<br>
At random (but consistently across attempts to train): DDP throws a SIGSEGV and exits with a SIGTERM when 33/97 batches have been evaluated or say 10/97 batches have been evaluated etc.,</p>
<p>At a loss why this is happening, plenty of GPU vRAM, local memory etc., is available – do not get any more information, it simply exits.</p>
<p><strong>EDIT</strong><br>
Additional Context: When I only run 1-2 batches of training, break and then run the test loop – it completes as expected. i.e. when one full epoch of training is run, and subsequently I run inference the exception occurs.</p>
<p>I have done the following:</p>
<ol>
<li>Experimented with the aforementioned pytorch versions/CUDA versions.</li>
<li>Machines: GCP and a local cluster (same hardware setup)</li>
</ol>
<p>My test function is very simple:</p>
<pre><code class="lang-auto">def test(self):
        self.model.eval()
        test_loss = 0
        correct = 0
        total = 0

        tloader = tqdm.tqdm(self.test_loader, desc='Testing')
        with torch.no_grad():
            for inputs, targets in tloader:
                with autocast(dtype=torch.bfloat16):
                    outputs = self.model(inputs)
                    loss = self.criterion(outputs, targets)

                test_loss += loss.item()
                _, predicted = outputs.max(1)
                total += targets.size(0)
                correct += predicted.eq(targets).sum().item()

        test_loss /= (len(self.test_loader))
        accuracy = 100. * correct / total

        return test_loss, accuracy
</code></pre>
<p>When I run this without torch.compile, all runs fine <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=12" title=":confused:" class="emoji" alt=":confused:" loading="lazy" width="20" height="20"></p>

username: sree_harsha
cooked: <p>Hi, I’d like to revisit this, do you have any thoughts <a class="mention" href="/u/ptrblck">@ptrblck</a>?</p>
<p>Training ResNet50 on ImageNet, batch size per gpu 128 on 4xA100 (using FFCV as the dataloader).</p>
<p>Few changes since this post:</p>
<ul>
<li>changed from mp.spawn to torchrun</li>
<li>updated the test loop</li>
</ul>
<p>(1)torch.compile(model, ‘reduce-overhead’) + DDP + torch 2.5 → slowly grows in memory and throws OOM and exits<br>
(2)<br>
torch.compile(model, ‘reduce-overhead’/‘max-autotune’) + DDP other versions (even 2.4 etc.,) → SIGTERM at inference<br>
File “/opt/conda/bin/torchrun”, line 33, in <br>
sys.exit(load_entry_point(‘torch==2.2.1’, ‘console_scripts’, ‘torchrun’)())<br>
But installed version is not 2.2.1…<br>
(3) DDP with (no torch.compile) → runs fine</p>
<p>additional information:</p>
<ul>
<li>I add a buffer at every layer, to act as a mask during the forward pass - before compiling of course and to ensure it works fine I added:</li>
<li>torch._dynamo.config.guard_nn_modules=True (so that the mask got updated)</li>
<li>Tried compiling before DDP and after, to same effect.</li>
</ul>
<p>After a few batches on the test set, it just dies…</p>
<p>Really not sure what to do here.</p>

username: ptrblck
cooked: <p>I don’t know what might be causing the issues and also don’t fully understand your use case as it seems you are trying to launch multiple DDP jobs at once inside this loop?</p>
<aside class="quote no-group" data-username="sree_harsha" data-post="1" data-topic="203462">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sree_harsha/48/14426_2.png" class="avatar"> sree_harsha:</div>
<blockquote>
<pre><code class="lang-auto">for i, threshold in enumerate(thresholds):
        mp.spawn(main, args=(init_model, world_size, threshold, i), nprocs=world_size, join=True)
</code></pre>
</blockquote>
</aside>
<aside class="quote no-group" data-username="sree_harsha" data-post="2" data-topic="203462">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sree_harsha/48/14426_2.png" class="avatar"> sree_harsha:</div>
<blockquote>
<p>File “/opt/conda/bin/torchrun”, line 33, in<br>
sys.exit(load_entry_point(‘torch==2.2.1’, ‘console_scripts’, ‘torchrun’)())<br>
But installed version is not 2.2.1…</p>
</blockquote>
</aside>
<p>I would recommend checking why this old PyTorch release is used e.g. by checking the <code>torch.__path__</code> making sure the right binary is used.</p>
<aside class="quote no-group" data-username="sree_harsha" data-post="2" data-topic="203462">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sree_harsha/48/14426_2.png" class="avatar"> sree_harsha:</div>
<blockquote>
<p>(1)torch.compile(model, ‘reduce-overhead’) + DDP + torch 2.5 → slowly grows in memory and throws OOM and exits</p>
</blockquote>
</aside>
<p>This is often a user error where e.g. the loss is accumulated or attached to a <code>list</code> thus disallowing PyTorch to delete intermdiate activations and causing an OOM eventually.</p>

username: sree_harsha
cooked: <p>Hi!</p>
<p>(1) <strong>Regarding use-case</strong><br>
My use case is iterative magnitude pruning.</p>
<ul>
<li>I initialize a model where each layer has a buffer for a mask, initially all set to 1.</li>
<li>The mask is applied at the forward pass, at level 0 – this has no impact (other than some overhead assumedly)</li>
<li>I train the dense model using DDP (just one process)</li>
<li>Then once complete, prune a % of the weights (set them to 0 in the corresponding buffer)</li>
<li>Release the distributed training group.</li>
<li>Initialize and start next level of training</li>
<li>Repeat till sparsity target is reached.<br>
The challenge initially was that torch.compile wasn’t detecting the change in the buffer, which is why I had to add <code>torch._dynamo.config.guard_nn_modules=True</code> at the start.</li>
</ul>
<p>So at any given point, I just spawn a single instance of DDP.</p>
<p>(2) Version</p>
<ul>
<li>I tried it 2-3 times, but I will try it with a different docker container which starts from scratch with 2.4/2.5 and will check – thanks!</li>
</ul>
<p>(3) Regarding the OOM<br>
Will look into it, in the train as well as test loop, I make sure to include only the <code>.item()</code> of the loss. But this could very well be the reason, my confusion also stemmed from the fact that this doesn’t happen on earlier versions of torch, and when torch.compile is not used.</p>
<p>(3) could be user-error, I’ll make changes to see what I’m doing wrong ^^</p>
<p>I’ll keep this updated, thanks for the suggestions!</p>

username: sree_harsha
cooked: <p>Okay I got it working with with PyTorch 2.4 –</p>
<p>The challenge was potentially how I was keeping track of the loss values in a tensor as I called all_reduce on it. This was _potentially_causing the blowup and subsequent OOM leading to a SIGTERM at inference. This has now been dealt with, I delete the tensor after every epoch.</p>
<p>Thanks for pointing that out.</p>
<p>However, I think the problem that memory use is growing with each batch processed seems to persist with PT 2.5, though I will run a few ablations to check if that is always the case.</p>
<p>In any case – appreciate the support!</p>