Gradient checkpointing Bayesian Neural Network

sebastienwood: <p>Hi,<br>
I am considering the use of gradient checkpointing to lessen the VRAM load. From what I understand there were some issues with stochastic nodes (e.g. Dropout) at some point in time to apply gradient checkpointing.<br>
However I have a kind of Bayesian Neural Network which needs quite a bit of memory, hence I am interested in gradient checkpointing. Since all the weights are bayesian, there is stochasticity everywhere which can’t be removed.<br>
Can I safely apply gradient checkpointing to this Bayesian Neural Network ?</p>

ptrblck: <p>Based on the description from the <a href="https://pytorch.org/docs/stable/checkpoint.html">docs</a> it seems the default behavior would be a deterministic dropout, which can be disabled:</p>
<blockquote>
<p>Checkpointing is implemented by rerunning a forward-pass segment for each checkpointed segment during backward. This can cause persistent states like the RNG state to be advanced than they would without checkpointing. By default, checkpointing includes logic to juggle the RNG state such that checkpointed passes making use of RNG (through dropout for example) have deterministic output as compared to non-checkpointed passes. The logic to stash and restore RNG states can incur a moderate performance hit depending on the runtime of checkpointed operations. If deterministic output compared to non-checkpointed passes is not required, supply  <code>preserve_rng_state=False</code>  to  <code>checkpoint</code>  or  <code>checkpoint_sequential</code>  to omit stashing and restoring the RNG state during each checkpoint.</p>
</blockquote>
<p>So it seems your use case should be fine.<br>
Could you link to the known issues with stochastic nodes? I remember there were some issues, but couldn’t find these issues quickly.</p>

sebastienwood: <p>Thanks for your reply ! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"><br>
I found the issue discussed in this tutorial, section “Handling a few special layers in checkpointing” : <a href="https://github.com/prigoyal/pytorch_memonger/blob/master/tutorial/Checkpointing_for_PyTorch_models.ipynb" rel="nofollow noopener">https://github.com/prigoyal/pytorch_memonger/blob/master/tutorial/Checkpointing_for_PyTorch_models.ipynb</a></p>

ptrblck: <p>Thanks for the link!<br>
While this notebook gives you a really good example of the usage, note that it’s a bit outdated by now and if I’m not mistaken, <a class="mention" href="/u/mcarilli">@mcarilli</a>’s <a href="https://github.com/pytorch/pytorch/pull/14253">PR</a> should have enabled the bitwise accuracy between standard models and checkpointed models.</p>
<p><a href="https://github.com/pytorch/pytorch/blob/41363b299a04a43ecb7520b39843d4c3a27c063a/test/test_utils.py#L204-L257">These tests</a> also should verify this behavior.</p>

sebastienwood: <p>Thank you very much for the informations ! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"> It worked pretty well from my first few experiments.<br>
Have a nice day <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>