Loss.backward()

username: s_n
cooked: <p>class Mymodel(nn.Module):<br>
def <strong>init</strong>(self:<br>
super(Mymodel, self).<strong>init</strong>()</p>
<pre><code>    self.conv1 = nn.Conv2d(...)
    self.conv2= Myconv(...)

def forward(self, x, conv2=False):
    
    if conv2:
        self.conv2.weight.data = self.conv1.weight.data    
        out  = self.conv2(x)
    else:
        out  = self.conv1(x)
    return out
</code></pre>
<p>model = Mymodel()<br>
output = model(input, conv2=True)<br>
In the above example, I want to do a forward pass in which my input is fed to conv2 instance and in backward pass, the weghts for conv1 needs to be updated. But when I do a loss.backward(), it is actually going to the backward function for conv2 instead I want it to go to backward function for conv1.</p>
<pre><code>Is it possible that when I do loss.backward() it goes through the backward function for conv1?</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>You should not use <code>.data</code> anymore. No valid use case for it exists.</p>
<blockquote>
<p>it is actually going to the backward function for conv2 instead I want it to go to backward function for conv1.</p>
</blockquote>
<p>The backward will match what happened during the forward pass. So if conv2 was used, it will be used during the backward.</p>

username: s_n
cooked: <p>Hi <a class="mention" href="/u/alband">@albanD</a><br>
Thanks for the reply!</p>
<p>conv2 (Myconv) is a class created by me using “Function” class in pytorch. Below is an example:</p>
<p>class Conv2d_mvm_function(Function):</p>
<pre><code>@staticmethod
def forward(ctx, input, weight, bias):
    ...
    
    ...
    ctx.save_for_backward(input, weight, bias)
    
@staticmethod
def backward(ctx, grad_output):
    
    input, weight, bias = ctx.saved_tensors
</code></pre>
<p>So when I do a loss.backward() it goes to line “ctx.saved_tensors” in my backward function, but I keep on getting this error “*** RuntimeError: No grad accumulator for a saved leaf!”</p>
<p>What is the reason for this error?<br>
The require grad for input and weight is True.</p>
<p>Also i am using double backward. Is there any problem with double backward?</p>
<p>What is the other way other than using *.data to make weights for self.conv2 to be equal to weights of self.conv1?</p>

username: albanD
cooked: <blockquote>
<p>No grad accumulator for a saved leaf!</p>
</blockquote>
<p>How do you get this error? Do you still use <code>.data</code>?<br>
Do you always have a bias? Does it require gradients? Do you return gradient in the backward for it all the time?</p>
<blockquote>
<p>What is the other way other than using *.data to make weights for self.conv2 to be equal to weights of self.conv1?</p>
</blockquote>
<p>Is there a reason why they are two different Tensors? If you just want them to have the same weight Tensor, just do that in the init by creating <code>MyConv</code> with <code>self.conv1.weight</code> or by passing this to your custom forward directly.</p>

username: s_n
cooked: <p>“No grad accumulator for a saved leaf!”<br>
I solved this error.  I made requires_gard=False for the parameters for which I didn’t want to calculate the grads</p>
<p>“What is the other way other than using *.data to make weights for self.conv2 to be equal to weights of self.conv1?”<br>
I have two tensors because I train with nn.conv2d and I infer with my conv implementation. What is the most efficient way for conv2 and conv1 to share weights for all the epochs. Basically the change in weight for conv1 should reflect in conv2 weight as well. I tried using the below line in my init:</p>
<p><strong>self.conv2.weight.data = self.conv1.weight.detach()</strong></p>
<p>But I think this would only make weight equal when init is called while initialising the model. The changes in conv1.weight won’t reflect on conv2 weights in subsequent epochs.</p>
<p>One way is to make the weight of conv2 and conv1 equal in forward function. But is there some other way to do so?</p>

username: albanD
cooked: <p>As I mentioned above, you just want a single Parameter for both to make sure they remain the same.<br>
Doing <code>self.conv2.weight = self.conv1.weight</code> will do that.</p>

username: s_n
cooked: <p>Is there a better way to make require_grad=False for all the parameters of conv2 in init method itself?</p>
<p>Currently I am making them False by individually making require grad=False for each parameter of the layer.</p>
<p>like self.conv2.weight=False<br>
self.bn2.weight = False<br>
…<br>
…</p>

username: albanD
cooked: <p>You can do <code>mod.requires_grad_(False)</code> on a nn.Module to set the requires grad field of all his parameters to False.</p>

username: s_n
cooked: <p>Hey <a class="mention" href="/u/alband">@albanD</a><br>
Will the below line allocate new memory for the weights for self.conv2 or it will just use the weights from the memory where self.conv1 weight is saved.<br>
" self.conv2.weight = self.conv1.weight"</p>

username: albanD
cooked: <p>Yes this will just set the <code>.weight</code> attribute to the Parameter contained in the other module. So they will point to the same Tensor.</p>

username: s_n
cooked: <p>So basically if any one of them (conv1.weight or conv2.weight) is updated it will be reflected in the other instance?</p>

username: albanD
cooked: <p>Yes it will be reflected.</p>

username: s_n
cooked: <p><a class="mention" href="/u/alband">@albanD</a> would both the options below make conv1 and conv2 share same weights?<br>
What I observed is that with option2 when I am updating conv1 it’s not reflected in conv2?<br>
Not sure what is the reason for that</p>
<p>Option1:<br>
class Mymodel1(nn.Module):<br>
def <strong>init</strong> (self:<br>
super(Mymodel, self). <strong>init</strong> ()</p>
<pre><code class="lang-auto">    self.conv1 = nn.Conv2d(...)
    self.conv2= nn.Conv2d(.....)
    self.conv2.weight = self.conv1.weight
</code></pre>
<p>Option2:<br>
class Mymodel2(nn.Module):<br>
def  <strong>init</strong> (self:<br>
super(Mymodel, self). <strong>init</strong> ()</p>
<pre><code class="lang-auto">    self.conv1 = nn.Conv2d(...)
    self.conv2= nn.Conv2d(...)

</code></pre>
<p>mymodel = Mymodel2()</p>
<p>mymodel.conv2.weight = mymodel.conv1.weight</p>
<p>In option2 when I did mymodel.conv2.weight.data = mymodel.conv1.weight  that actually helped to do what I was able to do with option1. But still I am curious why wouldn’t mymodel.conv2.weight =mymodel.conv1.weight won’t work in option2.</p>

username: albanD
cooked: <p>It should.<br>
Can you share a small script that shows how the two give different behavior?</p>