Coredump for pytorch ddp when job finished

kcsers: <p>Version pytorch 2.1.0<br>
I find that the WorkNccl will abort when job is finished, as shown below.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b314b1bef49b2cbb847f561cce53a070cfccac00.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/b314b1bef49b2cbb847f561cce53a070cfccac00" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b314b1bef49b2cbb847f561cce53a070cfccac00_2_690x195.jpeg" alt="image" data-base62-sha1="pydNSPuqO3myeSITS4BtxhhIRMY" width="690" height="195" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b314b1bef49b2cbb847f561cce53a070cfccac00_2_690x195.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b314b1bef49b2cbb847f561cce53a070cfccac00_2_1035x292.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/3/b314b1bef49b2cbb847f561cce53a070cfccac00_2_1380x390.jpeg 2x" data-dominant-color="080C07"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1920×543 109 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>FurtherMore, I find that the WorkNccl will not be synced or notified when the libcudart.so is unregietered. If the cuda device primary context is released, the WorkNccl  will core when it tries to destroy the cudaEvent.<br>
The key codes are listed</p>
<p>From the codes we find that Initializer in A is released first. Then the WorkNccl tries to destroy the CudaEvent. when it tries to set cuda device, TORCH_CHECK in B codes will fail.</p>
<p>The key is that WorkNccl does not check the device status when it tries to destroy the CUDAEvent inside.</p>

kcsers: <p>A. Initializer<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/2/92a88cfcdb6201a4e2bec906a568cf2ea085f27b.png" data-download-href="https://discuss.pytorch.org/uploads/default/92a88cfcdb6201a4e2bec906a568cf2ea085f27b" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92a88cfcdb6201a4e2bec906a568cf2ea085f27b_2_690x257.png" alt="image" data-base62-sha1="kVoPoxQMu1u19bUeku071yVqAzN" width="690" height="257" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92a88cfcdb6201a4e2bec906a568cf2ea085f27b_2_690x257.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/2/92a88cfcdb6201a4e2bec906a568cf2ea085f27b_2_1035x385.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/2/92a88cfcdb6201a4e2bec906a568cf2ea085f27b.png 2x" data-dominant-color="13181E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1114×416 37 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

kcsers: <p>B. Coredump codes<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/3/13440d3045e42e717b397871f87ab110af9af1ce.png" data-download-href="https://discuss.pytorch.org/uploads/default/13440d3045e42e717b397871f87ab110af9af1ce" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/3/13440d3045e42e717b397871f87ab110af9af1ce_2_690x284.png" alt="image" data-base62-sha1="2KqS2AWZOT46899aSMIkUemm8A6" width="690" height="284" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/3/13440d3045e42e717b397871f87ab110af9af1ce_2_690x284.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/3/13440d3045e42e717b397871f87ab110af9af1ce_2_1035x426.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/3/13440d3045e42e717b397871f87ab110af9af1ce.png 2x" data-dominant-color="F5F1F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1258×518 66.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

kcsers: <p>C. Error status<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/5/c/5cca2e8b20a54017c13da116fdcdc131a0fdc9b0.png" alt="image" data-base62-sha1="deR6bG4ZWYlkkMqnGiOqaEC2PkY" width="644" height="62"></p>

H-Huang: <p>Does this reproduce on the nightly version of pytorch? If so, could you create an issue on <a href="https://github.com/pytorch/pytorch/issues" rel="noopener nofollow ugc">github</a> and provide a reproducible script?</p>