Different forward and backward weights

Sia_Rezaei: <p>I have a use case, where I need to use a different set of weights to compute the backward pass. An instance of where this is used is in this work <a href="https://www.nature.com/articles/ncomms13276" rel="nofollow noopener">https://www.nature.com/articles/ncomms13276</a> and numerous follow up works. Or this <a href="https://www.siarez.com/projects/random-backpropogation" rel="nofollow noopener">https://www.siarez.com/projects/random-backpropogation</a></p>
<p>Is there anyway to do this with the current API? If not what would be the best angle of attack?</p>
<p>Thanks</p>

ptrblck: <p>You could try to load the backward <code>state_dict</code> before executing the <code>backward</code> operation, but it’s quite a hacky way:</p>
<pre><code class="lang-python">model = nn.Sequential(
    nn.Linear(1, 1, bias=False),
    nn.Linear(1, 1, bias=False)
)

with torch.no_grad():
    model[0].weight.fill_(1.)
    model[1].weight.fill_(1.)

sd_forward = copy.deepcopy(model.state_dict())
sd_backward = copy.deepcopy(sd_forward)
sd_backward['0.weight'].fill_(10.)
sd_backward['1.weight'].fill_(10.)


# one train step
output = model(torch.ones(1, 1))
model.load_state_dict(sd_backward)
output.mean().backward()
print(model[0].weight.grad)
&gt; tensor([[10.]])
print(model[1].weight.grad)
&gt; tensor([[1.]])
</code></pre>
<p>Also note, that the last gradient is wrong, since the output was calculated using the old weights.<br>
Would this approach work for you or did I misunderstand your question?</p>

Sia_Rezaei: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> Thanks for your reply. I actually figured out the right way to do this. I basically wrote my own Autograd <code>Function</code> class, similar to here: <a href="https://pytorch.org/docs/stable/notes/extending.html" rel="nofollow noopener">https://pytorch.org/docs/stable/notes/extending.html</a><br>
Then inside its <code>backward</code> method, I used a different set of weights to compute <code>grad_input</code>. So the backward now looks like this:</p>
<pre><code class="lang-auto">    def backward(ctx, grad_output):
        input, weight, b_weights, bias = ctx.saved_tensors
        grad_input = grad_weight = grad_bias = None
        if ctx.needs_input_grad[0]:
            grad_input = grad_output.mm(b_weights)
        if ctx.needs_input_grad[1]:
            grad_weight = grad_output.t().mm(input)
        if bias is not None and ctx.needs_input_grad[3]:
            grad_bias = grad_output.sum(0).squeeze(0)

        return grad_input, grad_weight, torch.zeros_like(b_weights), grad_bias
</code></pre>
<p><code>b_weights</code> are the backwards weights that are passed to the forward function and saved in <code>ctx</code></p>

ptrblck: <p>That looks like a good approach! Thanks for sharing it! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Hdk: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, I have a doubt, might be silly to ask🤐.<br>
But anyways, when we call loss.backward, will grad_input, grad_output, grad_weight and grad_bias become like these?</p>
<p>grad_input = gradient of loss w.r.t input,<br>
grad_output = gradient of loss w.r.t output,<br>
grad_weight = gradient of loss w.r.t weight,<br>
grad_bias = gradient of loss w.r.t bias.</p>
<p>And what do grad_input, grad_output, grad_weight and grad_bias mean as per this <a href="https://pytorch.org/docs/stable/notes/extending.html" rel="noopener nofollow ugc">tutorial</a>? Gradients of what and w.r.t. what?</p>

ptrblck: <p>The <code>weight</code> and <code>bias</code> gradients would be the gradients of the loss w.r.t these parameters.<br>
The input and output gradients are calculated through the chain rule to forward the gradient to the next layer (previous layer during the forward pass).</p>

Hdk: <p>I am getting confused because when I checked the documentations behind</p>
<pre><code class="lang-auto">grad_input = torch.nn.grad.conv2d_input(input.shape, weight, grad_output)
grad_weight = torch.nn.grad.conv2d_weight(input, weight.shape, grad_output)
</code></pre>
<p>I came to know the <code>input</code> is the input to the (convolution) layer and [this] (<a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/grad.py" rel="noopener nofollow ugc">https://github.com/pytorch/pytorch/blob/master/torch/nn/grad.py</a>) code file also says that <code>conv2d_weight</code> function (line 170) computes the gradient of output of convolution with respect to the weight of the convolution.</p>
<p>So, where does the gradient of loss w.r.t. parameters is calculated if you take for example <code>class LinearFunction(Function): in extending pytorch tutorial</code>?</p>

ptrblck: <p>I’m not sure if I understand the question properly, but you would apply the chain rule and thus the conv output would be used.<br>
The general workflow of the chain rule and backpropagation is explained e.g. in <a href="https://cs231n.github.io/optimization-2/">CS231n - Optimization</a>.</p>

Hdk: <p>Thank you for replying.</p>
<p>I am familiar with the chain rule, but I don’t know where exactly gradients of loss wr.t. parameters are calculated in code if I use autograd function as described in extending pytorch tutorial.</p>
<p>Can you please look at below code snippet in which I have commented my doubts, I think this would be a better way to clarify doubts <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"></p>
<pre><code class="lang-auto">class Custom_Convolution(torch.autograd.Function):    
    
    @staticmethod
    def forward(ctx, input, weight, bias, stride, padding):  # input's shape = ([batch_size=100, 96, 8, 8])
        output = torch.nn.functional.conv2d(input, weight, bias, stride, padding)  
        ctx.save_for_backward(input, weight, bias, output)
        return output    #output's shape = ([[batch_size= 100,128, 4, 4])
    
    @staticmethod
    def backward(ctx, grad_output):  #grad_output size = ([batch_size= 100, 128, 4, 4])    
        input, weight, bias, output = ctx.saved_tensors      #input's size = ([batch_size=100 , 96,8,8])
        print("op:",output.shape, output.requires_grad, output.grad_fn )
## It shows, output requires gradient and grad_fn = &lt;torch.autograd.function.Custom_ConvolutionBackward object at ...&gt;

## I am cloning the output because I think it will override the gradients 
## of already existing output tensor which may affect further calculations 
## of grad_input, grad_weight and grad_bias. 
##  PLEASE CORRECT ME IF I AM WRONG.

        features = output.clone()
        print("op2:",features.requires_grad, features.grad_fn )
## It prints: False , None  !!!!
## HOW CAN I RETAIN PAST HISTORY OF output SO THAT IT STILL 
## WOULD REQUIRE GRADIENT AND POSSESS A GRADIENT FUNCTION??

        features = features.view(features.shape[0], features.shape[1], -1)
        
        #Total_features=  features.shape[0]* features.shape[1]
                 
        cont_loss = torch.tensor([0.]).requires_grad_(requires_grad=True).to(dev)  # shape: ([1])
        for ..... :
                # My code for loss...   includes some operations like torch.div,exp,sum...
                # Calculation of loss for each feature 'i' : Li
                # cont_loss  +=  Li (Number of Li values = features.shape[0]* features.shape[1])

</code></pre>
<p>I want to backpropagate from cont_loss to features (i.e. output) and then features to  weight tensor.<br>
So, when I use <code> torch.autograd.grad(outputs= cont_loss, inputs= weight , retain_graph=(True))</code>,</p>
<p>I am getting RuntimeErrors like<br>
RuntimeError: One of the differentiated Tensors appears to not have been used in the graph. Set allow_unused=True if this is the desired behavior / One of the tensor used in computational graph either does not require gradient or it has No gradient function.</p>

ptrblck: <p>Based on your comments it seems you would like to apply something like a second order gradients, since you want to create <code>grad_fn</code>s inside the <code>backward</code>. If that’s the case, enable the gradient calculation via <code>with torch.enable_grad():</code>.</p>

Hdk: <p>Hi, I modified my code but <a href="https://discuss.pytorch.org/t/gradient-computation-in-custom-backward/102810/19">this</a> is happening.</p>