[BUG] {'use_reentrant': True} results in "Gradients will be None"
#74
by
RonanMcGovern
- opened
Seems there's no way to use reentrancy for gradient checkpointing without errors. This results in high memory for fine-tuning.
model.enable_input_require_grads() maybe adding this line could help.
I'm using the code with {'use_reentrant': True}
Ok many thanks, I'll try that on the next fine-tune. (although I don't understand exactly what that would fix things, seems it might, I just don't understand the theory).