Dynamo exceptions with distributeddataprallel / compile

thorstenwagner: <p>Hi,</p>
<p>I recently play with the compile option and distributed data parallel. Locally it workes like a charm, now I wanted to try it on our cluster and run into an exception that is related to dynamo:</p>
<pre><code class="lang-auto">  File "/raven/u/twagner/mambaforge/envs/tomotwin_pytorch2/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3795, in aot_module_simplified
    compiled_fn = create_aot_dispatcher_function(
  File "/raven/u/twagner/mambaforge/envs/tomotwin_pytorch2/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 194, in time_wrapper
    r = func(*args, **kwargs)
  File "/raven/u/twagner/mambaforge/envs/tomotwin_pytorch2/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 3333, in create_aot_dispatcher_function
    compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
  File "/raven/u/twagner/mambaforge/envs/tomotwin_pytorch2/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2120, in aot_wrapper_dedupe
    return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
  File "/raven/u/twagner/mambaforge/envs/tomotwin_pytorch2/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 2300, in aot_wrapper_synthetic_base
    return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
  File "/raven/u/twagner/mambaforge/envs/tomotwin_pytorch2/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1574, in aot_dispatch_base
    compiled_fw = compiler(fw_module, flat_args)
  File "/raven/u/twagner/mambaforge/envs/tomotwin_pytorch2/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 194, in time_wrapper
    r = func(*args, **kwargs)
  File "/raven/u/twagner/mambaforge/envs/tomotwin_pytorch2/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 1033, in fw_compiler_base
    return inner_compile(
  File "/raven/u/twagner/mambaforge/envs/tomotwin_pytorch2/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/raven/u/twagner/mambaforge/envs/tomotwin_pytorch2/lib/python3.10/site-packages/torch/_dynamo/repro/after_aot.py", line 80, in debug_wrapper
    inner_compiled_fn = compiler_fn(gm, example_inputs)
  File "/raven/u/twagner/mambaforge/envs/tomotwin_pytorch2/lib/python3.10/site-packages/torch/_inductor/debug.py", line 224, in inner
    return fn(*args, **kwargs)
  File "/raven/u/twagner/mambaforge/envs/tomotwin_pytorch2/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/raven/u/twagner/mambaforge/envs/tomotwin_pytorch2/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 52, in newFunction
    return old_func(*args, **kwargs)
  File "/raven/u/twagner/mambaforge/envs/tomotwin_pytorch2/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 312, in compile_fx_inner
    compiled_graph: CompiledFxGraph = fx_codegen_and_compile(
  File "/raven/u/twagner/mambaforge/envs/tomotwin_pytorch2/lib/python3.10/site-packages/torch/_inductor/compile_fx.py", line 534, in fx_codegen_and_compile
    compiled_fn = graph.compile_to_fn()
  File "/raven/u/twagner/mambaforge/envs/tomotwin_pytorch2/lib/python3.10/site-packages/torch/_inductor/graph.py", line 960, in compile_to_fn
    return self.compile_to_module().call
  File "/raven/u/twagner/mambaforge/envs/tomotwin_pytorch2/lib/python3.10/site-packages/torch/_dynamo/utils.py", line 194, in time_wrapper
    r = func(*args, **kwargs)
  File "/raven/u/twagner/mambaforge/envs/tomotwin_pytorch2/lib/python3.10/site-packages/torch/_inductor/graph.py", line 931, in compile_to_module
    mod = PyCodeCache.load_by_key_path(key, path, linemap=linemap)
  File "/raven/u/twagner/mambaforge/envs/tomotwin_pytorch2/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1068, in load_by_key_path
    exec(code, mod.__dict__, mod.__dict__)
  File "/tmp/torchinductor_twagner/fv/cfvfwaddcpvacy2mncxq7k727le6bt7dwj6uyrispgninqh36r6d.py", line 463, in &lt;module&gt;
    async_compile.wait(globals())
  File "/raven/u/twagner/mambaforge/envs/tomotwin_pytorch2/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1344, in wait
    scope[key] = result.result()
  File "/raven/u/twagner/mambaforge/envs/tomotwin_pytorch2/lib/python3.10/site-packages/torch/_inductor/codecache.py", line 1203, in result
    self.future.result()
  File "/raven/u/twagner/mambaforge/envs/tomotwin_pytorch2/lib/python3.10/concurrent/futures/_base.py", line 458, in result
    return self.__get_result()
  File "/raven/u/twagner/mambaforge/envs/tomotwin_pytorch2/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
torch._dynamo.exc.BackendCompilerFailed: backend='compile_fn' raised:
FileNotFoundError: [Errno 2] No such file or directory: 'ldconfig'

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information


You can suppress this exception and fall back to eager by setting:
    import torch._dynamo
    torch._dynamo.config.suppress_errors = True
</code></pre>
<p>Here is the code related to compile and distributeddataparallel:</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/MPI-Dortmund/tomotwin-cryoet/blob/pt2_distrdataparallel/tomotwin/modules/inference/embedor.py#L561">
  <header class="source">

      <a href="https://github.com/MPI-Dortmund/tomotwin-cryoet/blob/pt2_distrdataparallel/tomotwin/modules/inference/embedor.py#L561" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/MPI-Dortmund/tomotwin-cryoet/blob/pt2_distrdataparallel/tomotwin/modules/inference/embedor.py#L561" target="_blank" rel="noopener nofollow ugc">MPI-Dortmund/tomotwin-cryoet/blob/pt2_distrdataparallel/tomotwin/modules/inference/embedor.py#L561</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="551" style="counter-reset: li-counter 550 ;">
          <li></li>
          <li>    self.model = NetworkManager.create_network(self.tomotwin_config).get_model()</li>
          <li>    if checkpoint is not None:</li>
          <li>        try:</li>
          <li>            self.model.load_state_dict(checkpoint["model_state_dict"])</li>
          <li>        except RuntimeError:</li>
          <li>            print("Load before failed")</li>
          <li></li>
          <li>    self.model.to(self.rank)</li>
          <li></li>
          <li class="selected">    self.model = torch.compile(self.model, mode="reduce-overhead")</li>
          <li>    self.model = torch.nn.parallel.DistributedDataParallel(self.model, device_ids=[self.rank])</li>
          <li></li>
          <li>def embed(self, volume_data: VolumeDataset) -&gt; np.array:</li>
          <li>    """Calculates the embeddings. The volumes showed have the dimension NxBSxBSxBS"""</li>
          <li></li>
          <li>    # device = torch.device("cuda" if torch.cuda.is_available() else "cpu")</li>
          <li>    torch.backends.cudnn.benchmark = True</li>
          <li></li>
          <li>    dataset = TorchVolumeDataset(volumes=volume_data)</li>
          <li>    sampler_data = torch.utils.data.DistributedSampler(dataset, rank=self.rank, shuffle=True)</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I could now simply suppress the exception, but I first would like to understand what is happening.</p>
<p>Maybe someone could help me out?</p>
<p>Best,<br>
Thorsten</p>

Marjorie5: <p>Did you find the solution?</p>

thorstenwagner: <p>No, unfortunately not …</p>

thorstenwagner: <p>Even <a class="mention" href="/u/ptrblck">@ptrblck</a> has no idea? <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"></p>

thorstenwagner: <p>Ok, the problem was that ldconfig was not in the PATH.</p>
<p>Adding /sbin  to the path solved the problem.</p>
<p>Best,<br>
Thorsten</p>

Marjorie5: <p>Thanks for sharing the solution<a href="https://valerocreditcard.fun/" rel="noopener nofollow ugc">.</a>.</p>