When does loss.backward() get called when accumulating gradients?
Just wondering when backward gets called when you are accumulating over (say 8) batches. I had put a breakpoint in on_after_backward() and that seemed to be only getting called on the 8th iteration of training. According to this answer, in order to save on GPU memory, it seems its best to call loss.backward() on each iteration.
Dear @sachinruk,
This behaviour was a bug and should have been resolved on master and on_after_backward should be called after each backward call. In the case of accumulating over (say 8) batches, you should see on_after_backward  called 8 times on master, and only 1 time previously.
Best,
T.C