Checkpoint_sequential "Gradients will be None"?

username: Martin_cb
cooked: <p>I am having problems using the checkpoint_sequential function. Using the documentation, I define my model as</p>
<pre><code class="lang-auto">model = nn.Sequential(...)
input_var = checkpoint_sequential(model, chunks, input_var)
</code></pre>
<p>But when I start training I get the following error:</p>
<blockquote>
<p>“None of the inputs have requires_grad=True. Gradients will be None”</p>
</blockquote>
<p>I am guessing what may be happening is that it tries to checkpoint the input, which would not have a gradient. Is this a bug? Do I need to worry about it? Is there a workaround?</p>
<p>There was a discussion about this recently, but with no apparent resolution:</p><aside class="quote" data-post="16" data-topic="19117">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ychfan/48/19624_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/checkpoint-with-no-grad-requiring-inputs-problem/19117/16">Checkpoint with no grad requiring inputs PROBLEM</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    May I ask if there is any convenient solution for torch.utils.checkpoint.checkpoint_sequential? Thanks!
  </blockquote>
</aside>


username: ptrblck
cooked: <p><a href="https://discuss.pytorch.org/t/checkpoint-with-no-grad-requiring-inputs-problem/19117/9">This post</a> links to the workaround using a dummy tensor and <a href="https://discuss.pytorch.org/t/checkpoint-with-no-grad-requiring-inputs-problem/19117/11">this post</a> gives an example implementation. Is this not working for you?</p>

username: Martin_cb
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>  Thanks for the quick reply! I linked that thread in my OP, but I dont think there are any solutions there for checkpoint_sequential only for checkpoint.</p>
<p>Is this anything more than a cosmetic issue? Like does this break my backpropagation or just complains that my input has no gradients, so it cannot checkpoint that which it shouldn’t in the first place.</p>
<p>The error message is unclear:</p>
<blockquote>
<p>“UserWarning: None of the inputs have requires_grad=True. Gradients will be None”</p>
</blockquote>
<ul>
<li>does this apply to just the one layer (the input) which I dont care about or all subsequent layers?</li>
</ul>

username: ptrblck
cooked: <p>I assume no gradients will be calculated at all.<br>
You could run a quick test using a constant tensor, perform an update step, and compare the new output using the same input tensor. Both outputs should be equal, if the model was not updated.</p>
<p>The simple workaround for the <code>nn.Sequential</code> container would be to set the <code>requires_grad</code> attribute of the input to <code>True</code>, as the dummy input approach would be a bit more complicated.</p>

username: Martin_cb
cooked: <p>I just checked. Indeed backpropagation is broken.</p>
<p>Setting input.requires_grad=True, will make the warning go away and backprop work again that is true. However, calculating/storing the gradients for the Input should not be necessary as you dont need that for standard backprop and would also be a huge memory requirement as well, defeating the whole purpose of using checkpoint_sequential in the first place.</p>

username: ptrblck
cooked: <p>Based on the linked tutorial from the other post it seems to be a known limitation.<br>
How large is your input, that it would be a huge waste of memory?<br>
E.g. an input tensor of <code>[64, 3, 224, 224]</code> in <code>float32</code> would need ~36.75MB.</p>