Option for disable tf32
Hi, is there a way in trainer to disable tf32 for ampere architecture? It's motivated by this discussion:https://discuss.pytorch.org/t/numerical-error-on-a100-gpus/148032/2
cc @justusschock @kaushikb11 @awaelchli @Borda @rohitgr7
Hi @dnnspark! Simply setting the flags in your script doesn't work?
torch.backends.cuda.matmul.allow_tf32 = False
torch.backends.cudnn.allow_tf32 = False