Random cuda error at dist.barrier() after initialization before model creation

amsword: <p>the program is first run dist.init_process_group() and then sync between works by dist.barriers(). After these, it will create the data loader, model and then do the training. However, it crashes at dist.barrier() with random error message. Here is a few error. Most of the cases, the job can run well but this happens now and then.</p>
<ul>
<li>
<p>out of memory message<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/e/bef636991a51dbe724ab8f5a2c0aec521ecdf43a.png" data-download-href="https://discuss.pytorch.org/uploads/default/bef636991a51dbe724ab8f5a2c0aec521ecdf43a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/bef636991a51dbe724ab8f5a2c0aec521ecdf43a_2_690x287.png" alt="image" data-base62-sha1="rfkebRcHz8VnP07AwDE0Z2zbQLM" width="690" height="287" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/bef636991a51dbe724ab8f5a2c0aec521ecdf43a_2_690x287.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/bef636991a51dbe724ab8f5a2c0aec521ecdf43a_2_1035x430.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/e/bef636991a51dbe724ab8f5a2c0aec521ecdf43a_2_1380x574.png 2x" data-dominant-color="133F4A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2190×911 76.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</li>
<li>
<p>NCCL error<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0be9226a0e89d02b7b5fd700e6a60e40d5af06b5.png" data-download-href="https://discuss.pytorch.org/uploads/default/0be9226a0e89d02b7b5fd700e6a60e40d5af06b5" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/0/b/0be9226a0e89d02b7b5fd700e6a60e40d5af06b5.png" alt="image" data-base62-sha1="1HmJU3vd9L4zzOTuTpvOpJFJye1" width="690" height="290" data-dominant-color="143F4A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1926×811 61.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</li>
<li>
<p>illegal access<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/3/930123a44e69821c46ab073c37b31e98f921d957.png" data-download-href="https://discuss.pytorch.org/uploads/default/930123a44e69821c46ab073c37b31e98f921d957" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/3/930123a44e69821c46ab073c37b31e98f921d957.png" alt="image" data-base62-sha1="kYsCZD6aCSaclXS9UrpY4OQVFmD" width="690" height="301" data-dominant-color="133F4A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1896×828 63.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
</li>
</ul>

ptrblck: <p>The <code>dist.barrier()</code> would synchronize the code and could thus potentially reraise a valid exception.<br>
For the 3 different errors:</p>
<ul>
<li>reduce the model or batch size to avoid the OOM</li>
<li>the NCCL error might be a red herring and could in fact also be the OOM issue</li>
<li>Could you post the stack trace here, please? Also, are you using custom CUDA code?</li>
</ul>
<p>PS: you can post code snippets by wrapping them into three backticks ```, which makes debugging easier.</p>

amsword: <p>Thanks. Eventually, it is hardware issue.</p>