RuntimeError: CUDA error: device-side assert triggered Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions

username: Jyun
cooked: <p>Hello, This is my code with error in <strong>torch.manual_seed(myseed)</strong> and I have already added <strong>os.environ[‘CUDA_LAUNCH_BLOCKING’] = “1”</strong>. I don’t know how to fix it <img src="https://discuss.pytorch.org/images/emoji/apple/smiling_face_with_tear.png?v=12" title=":smiling_face_with_tear:" class="emoji" alt=":smiling_face_with_tear:" loading="lazy" width="20" height="20"></p>
<p>My code <img src="https://discuss.pytorch.org/images/emoji/apple/point_down.png?v=12" title=":point_down:" class="emoji" alt=":point_down:" loading="lazy" width="20" height="20"></p>
<p>myseed = cfg[‘seed’]  # set a random seed for reproducibility<br>
torch.backends.cudnn.deterministic = True<br>
torch.backends.cudnn.benchmark = False<br>
np.random.seed(myseed)<br>
torch.manual_seed(myseed)<br>
random.seed(myseed)<br>
if torch.cuda.is_available():<br>
torch.cuda.manual_seed_all(myseed)</p>
<p>save_path = os.path.join(cfg[‘save_dir’], cfg[‘exp_name’]) # create saving directory<br>
os.makedirs(save_path, exist_ok=True)</p>
<p>log_fw = open(f"{save_path}/log.txt", ‘w’) # open log file to save log outputs<br>
def log(text):     # define a logging function to trace the training process<br>
print(text)<br>
log_fw.write(str(text)+‘\n’)<br>
log_fw.flush()</p>
<p>log(cfg)  # log your configs to the log file</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/6/46c585bd14eb72490aa8b7b1543e0ee31e8f14ce.png" data-download-href="https://discuss.pytorch.org/uploads/default/46c585bd14eb72490aa8b7b1543e0ee31e8f14ce" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/6/46c585bd14eb72490aa8b7b1543e0ee31e8f14ce.png" alt="image" data-base62-sha1="a64BZr7d6qwpulba6b7Qehka7yC" width="690" height="460" data-dominant-color="252728"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1072×716 49 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <p>Are you getting the error directly when <code>torch.manual_seed(myseed)</code> is executed or are you running some code already beforehand?<br>
In the former case, was the setup working before and if so what changed?</p>

username: Jyun
cooked: <p>I just import some packages and set the config</p>
<p>configs <img src="https://discuss.pytorch.org/images/emoji/apple/point_down.png?v=12" title=":point_down:" class="emoji" alt=":point_down:" loading="lazy" width="20" height="20"><br>
cfg = {<br>
‘dataset_root’: ‘./Food-11’,<br>
‘save_dir’: ‘./outputs’,<br>
‘exp_name’: “simple_baseline”,<br>
‘batch_size’: 128,<br>
‘lr’: 1e-3,<br>
‘seed’: 20220013,<br>
‘loss_fn_type’: ‘KD’,<br>
‘weight_decay’: 1e-5,<br>
‘grad_norm_max’: 10,<br>
‘n_epochs’: 200,<br>
‘patience’: 40,<br>
}</p>
<p>However, I can run it on colab but  but not on my computer.<br>
Could this be a problem with my computer settings?</p>

username: ptrblck
cooked: <p>I don’t know how this config is used, but assuming the environment directly fails during the seeding without execution of any previous code I would suggest to delete the current environment and reinstall PyTorch into a new and empty virtual env.</p>