Bizarre InternalTorchDynamoError with locally and formerly working code

EIFY: <p>Hi, has anyone seen this error in the last 3 months or so running a compiled pytorch model with multi-GPU?</p>
<pre><code class="lang-auto">[rank0]: File "/usr/lib/python3/dist-packages/torch/_dynamo/variables/builder.py", line 529, in _wrap
[rank0]: if has_triton():
[rank0]: File "/usr/lib/python3/dist-packages/torch/utils/_triton.py", line 37, in has_triton
[rank0]: return is_device_compatible_with_triton() and has_triton_package()
[rank0]: File "/usr/lib/python3/dist-packages/torch/utils/_triton.py", line 33, in is_device_compatible_with_triton
[rank0]: if device_interface.is_available() and extra_check(device_interface):
[rank0]: File "/usr/lib/python3/dist-packages/torch/utils/_triton.py", line 23, in cuda_extra_check
[rank0]: return device_interface.Worker.get_device_properties().major &gt;= 7
[rank0]: File "/usr/lib/python3/dist-packages/torch/_dynamo/device_interface.py", line 191, in get_device_properties
[rank0]: return caching_worker_device_properties["cuda"][device]
[rank0]: torch._dynamo.exc.InternalTorchDynamoError: IndexError: list index out of range

[rank0]: from user code:
[rank0]: File "/usr/lib/python3/dist-packages/torch/_dynamo/external_utils.py", line 40, in inner
[rank0]: return fn(*args, **kwargs)

[rank0]: Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information
</code></pre>
<p>The same code was running fine last October-November on the same type of instance and still runs fine last week on my local machine. I have tried changing the version of PyTorch (2.4.1, 2.5.1, 2.6.0) and triton (3.2.0, 3.1.0, 3.0.0, and the last 2.x) but none of that helped. <code>torch.cuda.is_available()</code> and <code>torch.cuda.device_count()</code> look fine and simple test code (e.g. <a href="https://gist.github.com/EIFY/dee8c28e1f5c7d78388020abd36a627e" class="inline-onebox" rel="noopener nofollow ugc">MHA w/ mask on DDP · GitHub</a>) also works fine. What other libraries/drivers might contribute to this?</p>
<p>The error is probably very environment specific but the exact steps to reproduce on a <a href="https://lambdalabs.com/" rel="noopener nofollow ugc">Lambda</a> 8x A100 (40 GB SXM4) instance is as follows:</p>
<pre><code class="lang-auto">pip3 install --upgrade requests
pip3 install wandb
pip3 install schedulefree

git clone https://github.com/EIFY/mup-vit.git
cd mup-vit
NUMEXPR_MAX_THREADS=116 torchrun main.py --fake-data --batch-size 4096 --log-steps 100
</code></pre>

EIFY: <p><a href="https://github.com/vllm-project/vllm/issues/4069" class="inline-onebox" rel="noopener nofollow ugc">[Bug]: Cannot Load any model. IndexError with CUDA, multiple GPUs · Issue #4069 · vllm-project/vllm · GitHub</a> is the closest issue I can find but the author “Solved it with a fresh install with a new docker container” without nailing down the cause…</p>

ptrblck: <p>You are calling <code>main_worker</code> <a href="https://github.com/EIFY/mup-vit/blob/dbf80880306138d2e3251f10ea6a3746be9766c8/main.py#L222">here</a> with:</p>
<pre data-code-wrap="python"><code class="lang-python">args.gpu: None, args: Namespace(data='imagenet', workers=4, prefetch_factor=1, hidden_dim=384, input_resolution=224, patch_size=16, num_layers=12, num_heads=6, posemb='sincos2d', mlp_head=False, representation_size=None, pool_type='gap', register=0, epochs=90, log_steps=100, log_epoch=[], start_step=0, batch_size=4096, accum_freq=1, schedule_free=False, warmup=10000, lr=0.001, beta1=0.9, beta2=0.999, polynomial_weighting_power=0.0, decoupled_weight_decay=True, weight_decay=0.0001, grad_clip_norm=1.0, torchvision_inception_crop=False, lower_scale=0.05, upper_scale=1.0, mixup_alpha=0.2, randaug=True, randaug_magnitude=10, print_freq=100, resume='', evaluate=False, world_size=1, rank=-1, dist_url='env://', dist_backend='nccl', seed=None, gpu=None, multiprocessing_distributed=False, fake_data=True, logs='./logs/', name='2025_02_07-22_08_43-lr_0.001-b_4096', distributed=False, ngpus_per_node=8, checkpoint_path='./logs/2025_02_07-22_08_43-lr_0.001-b_4096/checkpoints')
</code></pre>
<p>thus setting the <code>args.gpu</code> to <code>None</code> causing the issue and I’m unsure how this code worked before.</p>

EIFY: <p>I just checked and locally on a single-GPU machine it still runs fine, either with torchrun or just plain python. So I am not sure what made it intolerant on a 8 GPU instance…</p>
<p>(All the additional arguments are to shrink the memory footprint and see some logs sooner)</p>
<pre><code class="lang-auto">$ torchrun main.py --fake-data --batch-size 1024 --accum-freq 8 --dist-backend gloo --log-steps 1
=&gt; Fake data is used!
Compiling model...
Test: [ 0/49]	Time  9.935 ( 9.935)	Loss 6.9078e+00 (6.9078e+00)	Acc@1   0.00 (  0.00)	Acc@5   0.02 (  0.00)
 *   Acc@1 0.001 Acc@5 0.005
Test: [ 0/49]	Time 11.767 (11.767)	Loss 6.9078e+00 (6.9078e+00)	Acc@1   0.00 (  0.00)	Acc@5   0.00 (  0.00)
 *   Acc@1 0.001 Acc@5 0.005
Test: [ 0/49]	Time  7.029 ( 7.029)	Loss 6.9078e+00 (6.9078e+00)	Acc@1   0.00 (  0.00)	Acc@5   0.01 (  0.01)
</code></pre>

ptrblck: <p>To launch <code>torchrun</code> on multiple devices you would use <code>torchrun --nproc_per_node==8 ...</code> which will then correspond to the <code>--local-rank</code> argument inside your script as described <a href="https://github.com/pytorch/pytorch/blob/91dfa829811dacadf77ecf082ae4cd5957ff2508/torch/distributed/run.py#L44-L61">here</a>.<br>
In your approach you are launching your script with <code>torchrun</code> only and are not using the <code>--local-rank</code> at all, so again unsure how this should have ever worked.<br>
Alternatively, you can also use a multiprocessing approach inside your script which will spawn the processes there as described in <a href="https://pytorch.org/tutorials/intermediate/dist_tuto.html">this tutorial</a>.</p>

EIFY: <p>Thanks for the explanation. The last time I ran it on a multi-GPU instance I think I have set <code>--multiprocessing-distributed</code>. Since without it it still runs locally I thought it should work. I will try it as soon as I grab an instance.</p>

EIFY: <p>Update: adding <code>--multiprocessing-distributed</code> back indeed fixed it. The error message could be less mysterious but ultimately the false alarm is on me. Thanks for the help!</p>