There doesn't seem to be a way to set requires_grad=True for parameters you don't want to update
Environment:
macOS
Python 3.8
pytorch-lightning 1.0.4
I know the title seems kind of self-contradictory, but I'm trying to train a GAN where I'm regularizing the loss with the gradient norm like in this paper. I'm unable to calculate the gradient penalty correctly because, for example, requires_grad for the discriminator is set to false when I'm evaluating the generator, even though I need to calculate gradients through both.
I've already tried setting up a third optimizer to handle the regularization, but it may lead to inaccurate results because the regularizer is optimizing separately, as opposed to being added to the loss function.
If anyone has been able to implement something like this with lightning, I would really like to know. This issue has been giving me grief for weeks at this point and I haven't been able to figure out a solution.