Freezing at "Using /home/user/.cache/torch_extensions as PyTorch extensions root..."

Jiqing_Zhang: <p>HELP！<br>
the program seems to just stop with once “Using /home/user/.cache/torch_extensions as PyTorch extensions root…” prints</p>

ptrblck: <p>Could you describe the use case and issue a bit more? When are you running into this hang and would it be possible to post a minimal code snippet to reproduce the issue?<br>
Also, could you update PyTorch to the latest release, if you are using an older one?</p>

Jiqing_Zhang: <p>Hi，as shown in the figure, there is no error message.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/b/abffba31ce32a1e570b70a3e83a4d3ef11b155f9.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/abffba31ce32a1e570b70a3e83a4d3ef11b155f9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/abffba31ce32a1e570b70a3e83a4d3ef11b155f9_2_690x63.jpeg" alt="image" data-base62-sha1="oxzxwtVTAL9DTMdU9CV5hcmGoWZ" width="690" height="63" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/abffba31ce32a1e570b70a3e83a4d3ef11b155f9_2_690x63.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/abffba31ce32a1e570b70a3e83a4d3ef11b155f9_2_1035x94.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/b/abffba31ce32a1e570b70a3e83a4d3ef11b155f9_2_1380x126.jpeg 2x" data-dominant-color="F4F4F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1734×160 77.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I use 3090, and the CUDA version is 11.1. I tried to update my pytorch from 1.7 to 1.9, it still hasn’t been resolved.</p>

ptrblck: <p>Thanks for the update. I don’t think you are seeing a hang, but just a runtime compilation of an extension.</p>

Jiqing_Zhang: <p>Hi， I waited for it 3.5 hours.  However, it still stops here. <img src="https://discuss.pytorch.org/images/emoji/apple/rofl.png?v=10" title=":rofl:" class="emoji" alt=":rofl:"></p>

ptrblck: <p>OK, that doesn’t sound right. <img src="https://discuss.pytorch.org/images/emoji/apple/stuck_out_tongue.png?v=10" title=":stuck_out_tongue:" class="emoji" alt=":stuck_out_tongue:"> Try to narrow down which extension is being built and then maybe run it in isolation (or pre-built it to check, if this would be working).</p>

googlebot: <p>there may be a leftover .lock file from aborted build, try cleaning either whole .cache or extension’s sub-directory, I don’t remember where this file appears, but I had a hang because of it</p>

Jiqing_Zhang: <p>I deleted the whole .cache, and it works now. <img src="https://discuss.pytorch.org/images/emoji/apple/speak_no_evil.png?v=10" title=":speak_no_evil:" class="emoji" alt=":speak_no_evil:"></p>

Rifle_Zhang: <p>deleting the /.cache/pytorch_extensions works for me.</p>

mustious: <p>Delete the torch_extension cache<br>
<code>rm -rf ~/.cache/torch_extensions/</code></p>
<p>Source: <a href="https://github.com/microsoft/DeepSpeed/issues/2816" class="inline-onebox" rel="noopener nofollow ugc">Training stuck after loading the model? · Issue #2816 · microsoft/DeepSpeed · GitHub</a></p>