Dataloader Crashes after few ecpochs

Yazdan_Salimi: <p>I recieve this error after few epochs during training on torch. I am using windows, cuda-toolkit version 12.2. it happens randomly during training.</p>
<p>Windows fatal exception: access violation</p>
<p>Current thread 0x000045e0 (most recent call first):<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\pkg_resources_vendor\packaging\version.py”, line 519 in _cmpkey<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\pkg_resources_vendor\packaging\version.py”, line 213 in <strong>init</strong><br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\utils\module.py”, line 562 in version_leq<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\utils\misc.py”, line 585 in is_module_ver_at_least<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\utils_pytorch_numpy_unification.py”, line 203 in floor_divide<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\utils_pytorch_numpy_unification.py”, line 223 in unravel_index<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\utils.py”, line 578 in generate_pos_neg_label_crop_centers<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\croppad\array.py”, line 1158 in randomize<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\croppad\dictionary.py”, line 982 in randomize<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\croppad\dictionary.py”, line 1000 in <strong>call</strong><br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\transform.py”, line 98 in _apply_transform<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\transform.py”, line 141 in apply_transform<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\compose.py”, line 111 in execute_compose<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\transforms\compose.py”, line 335 in <strong>call</strong><br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\data\dataset.py”, line 934 in _transform<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\data\dataset.py”, line 112 in <strong>getitem</strong><br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\torch\utils\data_utils\fetch.py”, line 51 in <br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\torch\utils\data_utils\fetch.py”, line 51 in fetch<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\torch\utils\data\dataloader.py”, line 674 in _next_data<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\torch\utils\data\dataloader.py”, line 630 in <strong>next</strong><br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\data\thread_buffer.py”, line 49 in enqueue_values<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\threading.py”, line 917 in run<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\threading.py”, line 980 in _bootstrap_inner<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\threading.py”, line 937 in _bootstrap</p>
<p>Thread 0x00003858 (most recent call first):<br>
</p>
<p>Thread 0x0000512c (most recent call first):<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\threading.py”, line 316 in wait<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\threading.py”, line 581 in wait<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\tqdm_monitor.py”, line 60 in run<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\threading.py”, line 980 in _bootstrap_inner<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\threading.py”, line 937 in _bootstrap</p>
<p>Main thread:<br>
Thread 0x00006a48 (most recent call first):<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\threading.py”, line 316 in wait<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\queue.py”, line 180 in get<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\data\thread_buffer.py”, line 76 in <strong>iter</strong><br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\data\thread_buffer.py”, line 99 in buffer_iterator<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\monai\data\thread_buffer.py”, line 199 in <strong>iter</strong><br>
File “D:\pyhton_files\source_codes_python\new-july-2023\yazdan.py”, line 4217 in train_regression_flexible<br>
File “D:\pyhton_files\source_codes_python\new-july-2023\yazdan.py”, line 4807 in easy_flexible_regression<br>
File “d:\pyhton_files\projects\lowdose_ct_recon\step#4—traintheregressionmodel.py”, line 87 in <br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\spyder_kernels\py3compat.py”, line 356 in compat_exec<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 473 in exec_code<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 615 in _exec_file<br>
File “C:\ProgramData\anaconda3\envs\monai\lib\site-packages\spyder_kernels\customize\spydercustomize.py”, line 528 in runfile<br>
File “C:\Users\Yazdan\AppData\Local\Temp\ipykernel_27192\612619163.py”, line 1 in </p>
<p>Restarting kernel…<br>
<strong>strong text</strong></p>
<p>Thank you for your help</p>