Changing the computed gradients before backprop
I want to add noise to the gradients in pytorch lightning . Specifically, something similar to this paper: https://arxiv.org/pdf/1511.06807.pdf . Basically, I would compute the gradients and before the call to backward, i want to add noise.
What is the best way to achieve this in pytorch lightning?
Dear @sebastiangonsal,
From your paper, it seems you might want to call backward which computes the gradients and then add some noise right ?
Therefore, you could override the before_optimizer_step and add noise to all the params.grad.
class TestModel(LightningModule):

    def on_before_optimizer_step(self):
        for param in self.parameters():
            param.grad += torch.rand_like(param.grad)

If you want this re-usable, you could just move this to a callback instead.