How to flag certain modules as non-deterministic
Hey,
Question: How can I set a module/layer in my model-class to always be non-deterministic (irrespective of the deterministic flag in pl.Trainer())?
Context: I use pl to train a simple AutoEncoder that uses bilinear upscaling in the decoder part. For debugging, I use the deterministic flag of the pl.Trainer(). However, I receive the following error
RuntimeError: upsample_bilinear2d_backward_out_cuda does not have a deterministic implementation, but you set 'torch.use_deterministic_algorithms(True)'. You can turn off determinism just for this operation if that's acceptable for your application. You can also file an issue at https://github.com/pytorch/pytorch/issues to help us prioritize adding deterministic support for this operation.

Unfortunately, the error does not hint at how to set the module to be non-deterministic and neither does the documentation.
Cheers
dsethz
Hi @dsethz!
The error comes from PyTorch, but not from Lightning, and I think it's not (shouldn't be) feasible even in pure PyTorch because the flag is for reproducibility and if you allow randomness in certain layers, you can't reproduce the same result anymore.
https://pytorch.org/docs/stable/notes/randomness.html