How to save a list of integers for backward when using CPP custom layer?

username: hasakii
cooked: <p>For a custom layer like that:</p>
<pre><code class="lang-python">
class CustomLayer(Function):
    @staticmethod
    def forward(ctx, input, int1, int2):
        ctx.save_for_backward(input, int1, int2)
        out = some_c_function(input, int1, int2)
        return out

    @staticmethod
    def backward(ctx, grad_output):
        input, int1, int2 = ctx.saved_tensors
        grad_input = torch.zeros_like(input)
        backward_func(input, grad_output, grad_input, int1, int2)
        return grad_input, None, None


</code></pre>
<p>But when I call this layer, error occurred:</p>
<pre><code class="lang-nohighlight">TypeError: save_for_backward can only save variables, but argument 3 is of type int
</code></pre>
<p>Now the question is, how can I save those integers for backward ?</p>

username: albanD
cooked: <p>Hi,</p>
<p>You need to use <code>save_for_backward</code> only for inputs and outputs.<br>
All other tensors or python objects can be saved in the <code>ctx</code> as <code>ctx.in1 = int1</code>.</p>

username: Blupon
cooked: <p><a class="mention" href="/u/alband">@albanD</a> why do we need to use <code>save_for_backwards</code> for input tensors only ? I just tried to pass one input tensor from <code>forward()</code> to <code>backward()</code> using <code>ctx.tensor = inputTensor</code> in <code>forward()</code> and <code>inputTensor = ctx.tensor</code> in <code>backward()</code> and it seemed to work.</p>
<p>I appreciate your answer since I’m currently trying to really understand when to use <code>ctx</code> rather than <code>save_for_backward</code> . <a href="https://discuss.pytorch.org/t/when-should-you-save-for-backward-vs-storing-in-ctx/6522/8">Another post</a> tells us memory leak <em>isn’t an issue any more</em> , and concludes on my question.</p>
<p>In addition, reading this other post suggests <em>“earlier torch versions it was not possible to <code>save_for_backward</code> an intermediate result, only inputs to the <code>forward</code> function, but that seems to be no longer the case.”</em></p>
<p>An actual limit would be the need for the saved value to be a variable (in the sense of pytorch tensor) in the case of <code>save_for_backward</code> .</p>

username: tom
cooked: <p>Oh! PyTorch internals time!</p>
<p>So what happens is that <code>save_for_backward</code> stores the tensors in <a href="https://github.com/pytorch/pytorch/blob/master/torch/csrc/autograd/saved_variable.h" rel="nofollow noopener"><code>SavedVariable</code></a>s. This is what does the sanity checks for the backward (<em>one of the variables needed for gradient computation has been modified by an inplace operation</em> being the perhaps most (in)famous).<br>
Now variables that aren’t input or output of your function (and don’t share storage…) are not visible to the outside world and thus cannot be changed in ways that would trigger those sanity checks.<br>
Now you can safely store other tensors, but SavedVariable won’t work for non-tensors.</p>
<p>Best regards</p>
<p>Thomas</p>

username: Blupon
cooked: <p>I see, sanity checks. Thanks ! <img src="https://discuss.pytorch.org/images/emoji/apple/smile.png?v=9" title=":smile:" class="emoji" alt=":smile:"></p>