StyleGAN2 Training Illegal Instruction on cache clear instruction

YM2132: <p>Hi all,</p>
<p>I am training a StyleGAN2 and am facing a multitude of crashes recently. I provide the python stack trace (I have anonymised the stack traces if the paths look slightly weird) through running the train.py with</p>
<pre><code class="lang-auto">python3 -X faulthandler -u StyleGAN2.py
</code></pre>
<p>One error is due to illegal instruction:</p>
<pre><code class="lang-auto"> 38%|███▊ | 591068/1562500 [29:32:52&lt;107:33:34, 2.51it/s]Fatal Python error: Illegal instruction

Thread 0x00000001 (most recent call first): Thread 0x00000002 (most recent call first): File "/path/to/python/lib/python3.11/threading.py", line 331 in wait File "/path/to/python/lib/python3.11/threading.py", line 629 in wait File "/path/to/python/lib/python3.11/site-packages/tqdm/_monitor.py", line 60 in run File "/path/to/python/lib/python3.11/threading.py", line 1045 in bootstrapinner File "/path/to/python/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00000003 (most recent call first): File "/path/to/python/lib/python3.11/threading.py", line 327 in wait File "/path/to/python/lib/python3.11/multiprocessing/queues.py", line 231 in _feed File "/path/to/python/lib/python3.11/threading.py", line 982 in run File "/path/to/python/lib/python3.11/threading.py", line 1045 in bootstrapinner File "/path/to/python/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00000004 (most recent call first): File "/path/to/python/lib/python3.11/selectors.py", line 415 in select File "/path/to/python/lib/python3.11/multiprocessing/connection.py", line 947 in wait File "/path/to/python/lib/python3.11/multiprocessing/connection.py", line 440 in _poll File "/path/to/python/lib/python3.11/multiprocessing/connection.py", line 257 in poll File "/path/to/python/lib/python3.11/multiprocessing/queues.py", line 113 in get File "/path/to/python/lib/python3.11/site-packages/torch/utils/data/_utils/pin_memory.py", line 32 in do_one_step File "/path/to/python/lib/python3.11/site-packages/torch/utils/data/_utils/pin_memory.py", line 55 in pinmemory_loop File "/path/to/python/lib/python3.11/threading.py", line 982 in run File "/path/to/python/lib/python3.11/threading.py", line 1045 in bootstrapinner File "/path/to/python/lib/python3.11/threading.py", line 1002 in _bootstrap

Current thread 0x00000005 (most recent call first): File "/path/to/python/lib/python3.11/site-packages/torch/cuda/memory.py", line 170 in empty_cache File "/path/to/project/StyleGAN2/StyleGAN2.py", line 1129 in &lt;module&gt;
</code></pre>
<p>With line 1129 being: torch.cuda.empty_cache()</p>
<p>Another error is a segfault</p>
<pre><code class="lang-auto">FID score for iteration 20000: 64.7917022705078175 [02:54&lt;00:00, 25.46it/s]
  1%|▏         | 20668/1562500 [3:51:10&lt;168:03:37,  2.55it/s]Fatal Python error: Segmentation fault

Current thread 0x00000001 (most recent call first):
  &lt;no Python frame&gt;

Thread 0x00000002 (most recent call first):
  File "/path/to/python/lib/python3.11/threading.py", line 331 in wait
  File "/path/to/python/lib/python3.11/threading.py", line 629 in wait
  File "/path/to/python/lib/python3.11/site-packages/tqdm/_monitor.py", line 60 in run
  File "/path/to/python/lib/python3.11/threading.py", line 1045 in bootstrapinner
  File "/path/to/python/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00000003 (most recent call first):
  File "/path/to/python/lib/python3.11/threading.py", line 327 in wait
  File "/path/to/python/lib/python3.11/multiprocessing/queues.py", line 231 in _feed
  File "/path/to/python/lib/python3.11/threading.py", line 982 in run
  File "/path/to/python/lib/python3.11/threading.py", line 1045 in bootstrapinner
  File "/path/to/python/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00000004 (most recent call first):
  File "/path/to/python/lib/python3.11/threading.py", line 327 in wait
  File "/path/to/python/lib/python3.11/multiprocessing/queues.py", line 231 in _feed
  File "/path/to/python/lib/python3.11/threading.py", line 982 in run
  File "/path/to/python/lib/python3.11/threading.py", line 1045 in bootstrapinner
  File "/path/to/python/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00000005 (most recent call first):
  File "/path/to/python/lib/python3.11/selectors.py", line 415 in select
  File "/path/to/python/lib/python3.11/multiprocessing/connection.py", line 947 in wait
  File "/path/to/python/lib/python3.11/multiprocessing/connection.py", line 440 in _poll
  File "/path/to/python/lib/python3.11/multiprocessing/connection.py", line 257 in poll
  File "/path/to/python/lib/python3.11/multiprocessing/queues.py", line 113 in get
  File "/path/to/python/lib/python3.11/site-packages/torch/utils/data/_utils/pin_memory.py", line 32 in do_one_step
  File "/path/to/python/lib/python3.11/site-packages/torch/utils/data/_utils/pin_memory.py", line 55 in pinmemory_loop
  File "/path/to/python/lib/python3.11/threading.py", line 982 in run
  File "/path/to/python/lib/python3.11/threading.py", line 1045 in bootstrapinner
  File "/path/to/python/lib/python3.11/threading.py", line 1002 in _bootstrap

Thread 0x00000006 (most recent call first):
  File "/path/to/python/lib/python3.11/site-packages/torch/autograd/graph.py", line 768 in enginerun_backward
  File "/path/to/python/lib/python3.11/site-packages/torch/autograd/init.py", line 289 in backward
  File "/path/to/python/lib/python3.11/site-packages/torch/_tensor.py", line 521 in backward
  File "/path/to/project/StyleGAN2/StyleGAN2.py", line 1093 in module
</code></pre>
<p>where line 1093 being: d_optimizer.step()</p>
<p>These errors are seemingly random, cropping up throughout training at different points.</p>
<p>Thanks for reading this post, I’d be glad to take any suggestions on how to fix <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>