question: Changing the computed gradients before backprop
context: I want to add noise to the gradients in pytorch lightning . Specifically, something similar to this paper: https://arxiv.org/pdf/1511.06807.pdf . Basically, I would compute the gradients and before the call to backward, i want to add noise.
  What is the best way to achieve this in pytorch lightning?
answer: Dear @sebastiangonsal,
  From your paper, it seems you might want to call backward which computes the gradients and then add some noise right ?
  Therefore, you could override the before_optimizer_step and add noise to all the params.grad.
  class TestModel(LightningModule):
  
      def on_before_optimizer_step(self):
          for param in self.parameters():
              param.grad += torch.rand_like(param.grad)
  
  If you want this re-usable, you could just move this to a callback instead.