How to init parameter of customized layer?

Ardeal: <p>Hi,</p>
<p>the following code is my customized layer.<br>
In one of my network, I used the customized layer MSConved and torch.nn.Conv2d in one of my network.</p>
<p>at present, I would like use the torch.nn.Conv2d.weight to init the self.weight of my MScon2d.weight, how to do this?</p>
<p>as I asked in another topic: <a href="https://discuss.pytorch.org/t/nn-parameter-re-define-question/129523" class="inline-onebox">Nn.parameter re-define question</a><br>
could parameter in customized layer be re-assigned?</p>
<p>is there any doc about the difference among parameter, tensor and variable in pytorch?</p>
<pre><code class="lang-python">import os
import numpy as np 
import torch
import torch.nn as nn
import torch.nn.functional as F
from torch.nn.modules.utils import _pair
from torch.nn.common_types import _size_2_t

class MSConv2d(nn.modules.conv._ConvNd):
    def __init__(self, in_channels: int, out_channels: int, kernel_size: _size_2_t, stride: _size_2_t = 1 ) -&gt; None:
        super(MSConv2d, self).__init__(in_channels, out_channels, kernel_size, stride)

        self.in_channels = in_channels
        self.out_channels = out_channels    
        self.kernel_size = _pair(kernel_size)
        
        self.weight = nn.Parameter(torch.empty((out_channels, in_channels, *self.kernel_size)))
        self.gamma = nn.Parameter(torch.empty((out_channels,self.num_groups,1,*self.kernel_size)))
        # nn.init.xavier_normal_(self.weight)
        # nn.init.constant_(self.gamma,1)

    def forward(self, x):
        # self.weight = self.weight*self.gamma
        ret = F.conv2d(F.pad(input), self.weight*self.gamma)
        return ret


</code></pre>

ptrblck: <aside class="quote no-group" data-username="Ardeal" data-post="1" data-topic="131476">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ardeal/48/33257_2.png" class="avatar"> Ardeal:</div>
<blockquote>
<p>at present, I would like use the torch.nn.Conv2d.weight to init the self.weight of my MScon2d.weight, how to do this?</p>
</blockquote>
</aside>
<p>You could assign the values via:</p>
<pre data-code-wrap="python"><code class="lang-python">with torch.no_grad():
    self.weight.copy_(conv_layer.weight)
</code></pre>
<aside class="quote no-group" data-username="Ardeal" data-post="1" data-topic="131476">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ardeal/48/33257_2.png" class="avatar"> Ardeal:</div>
<blockquote>
<p>is there any doc about the difference among parameter, tensor and variable in pytorch?</p>
</blockquote>
</aside>
<p>I think <a href="https://pytorch.org/tutorials/beginner/pytorch_with_examples.html">this tutorial</a> could be helpful.</p>

Ardeal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
Thank you!</p>
<aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="131476">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<pre><code class="lang-auto">with torch.no_grad():
    self.weight.copy_(conv_layer.weight)
</code></pre>
</blockquote>
</aside>
<p>This code solve my issue. Thank you!</p>
<p>I read the link you pasted above. but it doesn’t answer my question:<br>
whare is the difference of parameter, tensor and variable? is there any doc about that?</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> , Could you please kindly answer the qestion I asked in the link: <a href="https://discuss.pytorch.org/t/nn-parameter-re-define-question/129523" class="inline-onebox">Nn.parameter re-define question</a></p>

ptrblck: <p>The short answer is: <code>Variable</code>s are deprecated, <code>nn.Parameters</code> require gradients and are automatically registered inside <code>nn.Module</code>s (so that they are returned in their <code>state_dict</code> and pushed to the device via <code>model.to()</code>), tensors are just array-like objects which can require gradients.</p>

Ardeal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> ，<br>
Many thanks！<br>
You  answer confirmed my understanding.<br>
paramerter:</p>
<ol>
<li>will be added to state_dict, and named_parameters()</li>
<li>will be pushed to device, will tensor be pushed to device via model.to()? do you mean that tensor should be pushed to device manually by tensor.to()?</li>
</ol>
<ol>
<li>
<code>self.mask = self.mask_weight*temp</code>, by this line, could I keep self.mask_weght and self.mask are in the same memory? will they be upated in sync?</li>
<li>why could I not be able to <code>self.mask_weight = self.mask_weight * temp</code> to re-assign the parameter?</li>
<li>does it mean that nn.parameter could only be defined once, and could not be re-defined?</li>
</ol>
<p>I got a few quesiton about parameter assignment, could you please kindly answer me?</p>
<pre><code class="lang-python">class SMConv2d(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, padding=1, stride=1):
        super(SMConv2d, self).__init__()
    
        self.mask_weight = nn.Parameter(torch.Tensor(out_channels, in_channels, kernel_size, kernel_size))
        nn.init.constant_(self.mask_weight, 1)
    
    def compute_mask(self,temp):
        self.mask = self.mask_weight*temp
        # self.mask_weight = self.mask_weight * temp
       
        return self.mask 

    def forward(self, x, temp=1):
        masked_weight = self.compute_mask(temp)
        out = F.conv2d(x, masked_weight, stride=self.stride, padding=self.padding)        
        return out

</code></pre>

Ardeal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p><code> self.mask_weight = self.mask_weight * temp </code></p>
<p>Could parameter in customized layers be re-assigned likt the above  code?</p>

ptrblck: <p>You can reassign parameters, but these will be new parameters and thus not updated anymore.<br>
If you want to manipulate the parameter use:</p>
<pre><code class="lang-python">with torch.no_grad():
    self.mask_weight.copy_(...)
</code></pre>
<p>instead.</p>

Ardeal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>Thank you!</p>
<p>in customized layer, parameter should not be re-assigned by =.<br>
during model loading and initialization, if I use model.named_parameters() to get each parameter, could I use “=” to re-initialize it?</p>

ptrblck: <p>The same reasoning applies here as well:<br>
you can re-assign new parameters, but the internal parameter (assigned to <code>self.param</code>) is a <em>new</em> parameter and in case you have already passed <code>model.parameters()</code> to the optimizers etc. this <em>new</em> parameter will not be trained.<br>
It depends on your use case if that’s a concern or not.</p>

Ardeal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> ,<br>
Thank you!</p>
<p>in summarization:<br>
if parameter is re-assigned in layer or out of layer, parameter will not be parameter and will become tensor.  and will not be optimzed</p>

ptrblck: <p>No, this is not correct. The new parameter, will still be a parameter, but since it’s a new object you have to be careful where the old object was already used (e.g. via a reference in the optimizer) and would thus need to update it etc.<br>
The critical point is that you have to consider that the reassignment creates a <em>new</em> parameter (object) and then you should check if any other changes are needed (e.g. adding it to the optimizer).</p>

Ardeal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> ,  Thank you</p>
<p>I tried the following example code.<br>
<code> self.mask_weight = self.mask_weight * temp </code> # here, temp is a value, such float temp=3.0</p>
<p>for the upper code, I experienced the error below:</p>
<pre><code class="lang-python">cannot assign 'torch.cuda.FloatTensor' as parameter 'mask_weight' (torch.nn.Parameter or None expected)
</code></pre>
<pre><code class="lang-python">class SMConv2d(nn.Module):
    def __init__(self, in_channels, out_channels, kernel_size, padding=1, stride=1):
        super(SMConv2d, self).__init__()
    
        self.mask_weight = nn.Parameter(torch.Tensor(out_channels, in_channels, kernel_size, kernel_size))
        nn.init.constant_(self.mask_weight, 1)
    
    def compute_mask(self,temp):
        self.mask = self.mask_weight*temp
        # self.mask_weight = self.mask_weight * temp
       
        return self.mask 

    def forward(self, x, temp=1):
        masked_weight = self.compute_mask(temp)
        out = F.conv2d(x, masked_weight, stride=self.stride, padding=self.padding)        
        return out
</code></pre>

ptrblck: <p>As described in the error message, your would need to assign a new <code>nn.Parameter</code> to <code>self.mask_weight</code>, if you really want to replace it:</p>
<pre><code class="lang-python">self.mask_weight = nn.Parameter(self.mask_weight * temp)
</code></pre>

Ardeal: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> ,</p>
<p>Thank you!</p>
<aside class="quote no-group" data-username="ptrblck" data-post="13" data-topic="131476">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p><code>self.mask_weight = nn.Parameter(self.mask_weight * temp)</code></p>
</blockquote>
</aside>
<p>the upper code will generate a new parameter <code>self.mask_weight</code>. so I need to consider whether the new <code>self.mask_weight </code>is added to optimizer. is my understanding correct?</p>
<p>once I do <code>self.mask_weight * temp</code>,  the code return a tensor but not a parameter.  If I want to assign it parameter, I should do <code>self.mask_weight = nn.Parameter(self.mask_weight * temp)</code>. right?</p>