Problem with autograd.Function

335621: <p>I used to adopt a self-defined Function using PyTorch 1.1.0.<br>
Now a newer PyTorch (1.10.0) said the function must be used with a static method.<br>
I learn the tutorial but still don’t know how to make the function maintain a variable (here is <code>self.V</code>) during the forward and backward. Anyone could help?</p>
<pre><code class="lang-auto">class Exclusive(autograd.Function):
    def __init__(self, V):
        super(Exclusive, self).__init__()
        self.V = V

    def forward(self, inputs, targets):
        self.save_for_backward(inputs, targets)
        outputs = inputs.mm(self.V.t())
        return outputs

    def backward(self, grad_outputs):
        inputs, targets = self.saved_tensors
        grad_inputs = grad_outputs.mm(self.V) if self.needs_input_grad[0] else None
        for x, y in zip(inputs, targets):
            self.V[y] = F.normalize( (self.V[y] + x) / 2, p=2, dim=0)
        return grad_inputs, None
</code></pre>

KFrank: <p>Hi 3!</p>
<aside class="quote no-group" data-username="335621" data-post="1" data-topic="167022" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/3/c0e974/48.png" class="avatar"> 335621:</div>
<blockquote>
<p>how to make the function maintain a variable (here is <code>self.V</code>) during the forward and backward.</p>
</blockquote>
</aside>
<p>As you’ve recognized, because pytorch now requires the <code>forward()</code> and<br>
<code>backward()</code> functions of an <code>autograd.Function</code> to be static, you can no<br>
longer store any state (your <code>self.V</code>) in an instance of your <code>Exclusive</code>.</p>
<p>For this reason, pytorch has added a <em>context</em> argument (<code>ctx)</code> to <code>forward()</code><br>
and <code>backward()</code> that is used to save information from the <code>forward()</code> call<br>
for use in the <code>backward()</code> call.</p>
<p>So you could do something like:</p>
<pre data-code-wrap="python"><code class="lang-python">    @staticmethod
    def forward (ctx, V, inputs, targets):
        ctx.save_for_backward (V, inputs, targets)
        ...
    
    @staticmethod
    def backward (ctx, grad_outputs):
        V, inputs, targets = ctx.saved_tensors
        ...
</code></pre>
<p>You would have to store <code>V</code> somewhere else in your model, and then pass it<br>
in when you call, e.g., <code>Exclusive.apply (my_model.V, inputs, targets)</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

335621: <p>Dear Frank,</p>
<p>Thank you so much for your kind help!<br>
I think besides the <code>ctx</code>, I need to set the return of <code>backward()</code> to be</p>
<pre><code class="lang-auto">        return grad_inputs, None, None 
</code></pre>
<p>Below is the full code:</p>
<pre><code class="lang-auto">class Exclusive(autograd.Function):



    @staticmethod
    def forward(ctx, V, inputs, targets):
        ctx.save_for_backward(V, inputs, targets)
        outputs = inputs.mm(V.t())
        return outputs

    @staticmethod
    def backward(ctx, grad_outputs):
        V, inputs, targets = ctx.saved_tensors
        grad_inputs = grad_outputs.mm(V) if ctx.needs_input_grad[0] else None
        for x, y in zip(inputs, targets):
            V[y] = F.normalize( (V[y] + x) / 2, p=2, dim=0)
        return grad_inputs, None, None

</code></pre>
<p>Everything is OK now!<br>
Thank you!</p>