Defining weights of a custom layer as parameters

username: sriharsha0806
cooked: <pre><code class="lang-auto">class Mask(nn.Module):
	def __init__(self):
		super(Mask, self).__init__()
		self.weight = torch.nn.Parameter(data=torch.Tensor(outC, inC, kernel_size, 
			kernel_size), requires_grad=True)
		stddev = 1.0/math.sqrt(inC*kernel_size*kernel_size)
		self.weight.data.uniform(-stddev, stddev)
		self.weight = self.weight.contiguous()
		self.func()

	def func(self):
		...

	def forward(self, inp):
		masked_wt = self.weight.mul(self.mask.cuda())
		return torch.nn.functional.Conv2d(inp, masked_wt, stride=self.stride, padding=self.padding)



class Model(nn.Module):
	def __init__(self, inC=3, outC=32, kernel_size=1, stride=1, padding=0,
		groups=2, bisa=None):
		self.Mask = Mask(inC, outC, kernel_size, stride, groups=groups, bias=bias)
	def forward(self,x):
		x = Mask(x)
		return x
</code></pre>
<p>I Intialised Mask.weight as a nn.Parameter. I’m using this module in a larger network. After training when i print all the parameters using named_parameters() iterator. The custom layer weights are not appearing.</p>

username: ptrblck
cooked: <p>I’ve deleted some unnecessary parts of your model and just checked, if <code>Mask.weight</code> are in the parameters, and it seems to work.<br>
Also, I’ve formatted the code in your post. You can post code enclosed in three `.</p>
<pre><code class="lang-auto">class Mask(nn.Module):
    def __init__(self):
        super(Mask, self).__init__()
        self.weight = torch.nn.Parameter(data=torch.Tensor(1, 1, 1, 1), requires_grad=True)
        
        self.weight.data.uniform_(-1, 1)
        
    
    def forward(self, x):
        masked_wt = self.weight.mul(1)
        return masked_wt


class Model(nn.Module):
    def __init__(self):
        super(Model, self).__init__()
        self.Mask = Mask()

    def forward(self,x):
        x = Mask(x)
        return x

model = Model()

for name, param in model.named_parameters():
    print(name, param)
</code></pre>

username: sriharsha0806
cooked: <p>Thanks for the help these past few weeks. But I still have a doubt. I built Model A and Model B of MobileNet. Model A is build using Mask for pointwise convolutions of groups 2  which is Dense and Model B is build  using nn.Conv2d of groups 2. I tried to pass a random Tensor [1,3,32,32] to Model A and copied the weights of Model A to Model B [[1,0; 0, 1] --&gt; [1;1] in this fashion. I am getting same output for both models. I loaded the trained weights and it is still giving the same error.</p>
<p>I thought validation accuracy should be same for both. but I am getting for Model A 85.9% and for Model B 10% on cifar-10 dataset.</p>
<p>When I disabled Net.eval(), I’m getting same Validation Accuracy for both Models 81.3%.</p>
<p>Case II:<br>
I passed random Tensor as input to Models with ModelA.eval() and ModelB.eval() statements after loading weights from state_dict. I’m getting different outputs for both models compared to outputs for ModelA and ModelB which are same before initialising ModelA and ModelB.</p>
<p>What is the reason behind this? Can I still say ModelA and ModelB are behaving the same?</p>
<h1>Update</h1>
<p>Currently removing BatchNorm weights and training the model. Will validate the model and let you know. But I do not understand why BatchNorm should cause this error.</p>

username: ptrblck
cooked: <p>As far as I understand, you are somehow copying weights between modelA and modelB. Was one model trained and the other randomly initialized?<br>
<code>BatchNorm</code> layers come with weights and a bias (gamma and beta in the paper) as well as with the running statistics (<code>running_mean</code> and <code>running_var</code>).<br>
If you forget to copy the running stats, this could be the issue.</p>

username: sriharsha0806
cooked: <p>You’re correct.  I’m not copying running_mean and running_var to Model B.</p>
<p>I am storing weights as mentioned in pytorch imagenet tutorials.</p>
<pre><code class="lang-auto">'state_dict': model.state_dict()
</code></pre>
<p>I wrote print statement after prec=validate(testloader, model, criterion)</p>
<pre><code class="lang-auto">print(model.trianing)
</code></pre>
<p>It returned false.</p>
<p><span class="hashtag">#2</span></p>
<p>I loaded weights to ModelA as given in Tutorials</p>
<pre><code class="lang-auto">checkpoint = torch.load('./result/Mobilenet/grad_best.pth.tar')
Dict = checkpoint['state_dict']
Model.load_state_dict(Dict)
</code></pre>
<p>I’m copying weights from Model A to Model B<br>
batchnorm weights</p>
<pre><code class="lang-auto">Net.model[i][1].weight.data.copy_(Model.model[i][1].weight.data)
Net.model[i][1].bias.data.copy_(Model.model[i][1].bias.data)
</code></pre>
<h1>Question</h1>
<ol>
<li>
<p>Does state_dict() save running statistics? If not how to save running batchnorm weights?</p>
</li>
<li>
<p>When I’m copying batchnorm weights layer wise using <code>copy_</code> does it copy running batchnorm weights also. If not how to copy runninng batchnorm weights to Model B?</p>
</li>
</ol>

username: RyanCV
cooked: <p>Hi ptrblck, for the mask layer, self.weight has shape of (1,1,1,1). For my application, I want the mask be (Batch, Channel, Height, Width)=(32, 1, 28, 28), but the values of mask (actually the shape of mask is 1x28x28) in each batch (all 32 batches) are identical.<br>
question1: I want to update the mask values at each iteration using back propagation just like the mini-batch SGD to update the parameters. I tried to using torch.cat() to get 32 identical copies of mask, but it seems too silly way. Could you tell me how to implement it?</p>
<p>question2: how can it guarantee that only the mask is updated during the back propagation?<br>
Thanks.</p>

username: ptrblck
cooked: <p>I think you can just add a batch dimension of 1 into your mask and let broadcasting do the magic. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=6" title=":wink:" class="emoji" alt=":wink:"><br>
If you don’t want the gradient to be backpropagated to the input, you might need to detach it with <code>mask * x.detach()</code>.</p>

username: RyanCV
cooked: <p>for broadcasting, I didn’t find any function in torch to this, could you tell me which function do you mean for broadcasting?</p>

username: Steve_Hu
cooked: <p>if i have defined a mask layer like this, how should i add it to a existed torchvision model,say add mask behind resnet50 some layer?</p>

username: ptrblck
cooked: <p>It depends a bit on your use case, but I think the most flexible and general use case would be to derive a custom class from resnet50 and to override the <code>forward</code> method with the mask layer applied to the appropriate layer.</p>
<p>Let me know, if this workflow would work for you or if you plan to apply the mask in another way.</p>

username: Steve_Hu
cooked: <p>it sounds a feasible idea, i will try it ! thanks:grin:</p>

username: Steve_Hu
cooked: <p>the hook technique cann’t change the forward value? right?</p>

username: ptrblck
cooked: <p>It should be possible, if you don’t detach the tensor.<br>
Are you seeing any issues using forward hooks?</p>

username: Steve_Hu
cooked: <p>nope，so i think use forward hook to add attention after some conv layer seems feasible and quick;</p>

username: Rowing0914
cooked: <p>I guess, this post might be related to our discussion!</p>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/58946855/pytorch-customize-weight/58961674#58961674" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/11744541/nicholashz" target="_blank" rel="nofollow noopener">
    <img alt="nicholashz" src="https://www.gravatar.com/avatar/05d36142414753e54a2567d82a147ef4?s=128&amp;d=identicon&amp;r=PG&amp;f=1" class="thumbnail onebox-avatar" width="60" height="60">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/58946855/pytorch-customize-weight/58961674#58961674" target="_blank" rel="nofollow noopener">Pytorch customize weight</a>
</h4>

<div class="tags">
  <strong>pytorch</strong>
</div>

<div class="date">
  
  answered by
  <a href="https://stackoverflow.com/users/11744541/nicholashz" target="_blank" rel="nofollow noopener">
    nicholashz
  </a>
  on <a href="https://stackoverflow.com/questions/58946855/pytorch-customize-weight/58961674#58961674" target="_blank" rel="nofollow noopener">07:08PM - 20 Nov 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


username: Rami_Nasser
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="17687">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>initia</p>
</blockquote>
</aside>
<p>in this setup the weights tensor have to be initialized with some values ?</p>

username: ptrblck
cooked: <p>If you are working with trainable parameters, then you would usually initialize them using a method from <code>nn.init</code> or any other random distribution.</p>
<p>I’m not sure, if I understand the question correctly, so please feel free to add more information in case I missed the point. <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=9" title=":wink:" class="emoji" alt=":wink:"></p>

username: Rami_Nasser
cooked: <p>I want to build a custom layer using a Parameter object, the layer applies some matrix multiplications on the input using the Parameter object(see below part of the code), my question is: do I have to initialize the Parameter with values in the constructor ? or it will implicitly be handled by PyTorch ?</p>
<pre><code class="lang-auto">class MuLayer(nn.Module):

    def __init__(self, comp, features):
        super(MuLayer, self).__init__()
        self.w = torch.nn.Parameter(data=torch.Tensor(comp, features), requires_grad=True)

    def forward(self, y, x):
        x.mm(torch.transpose(self.w, 0, 1))
        ...
</code></pre>

username: ptrblck
cooked: <p>You should either use a factory method (e.g. <code>torch.randn(size)</code>, which would create a tensor with values samples from the normal distribution) or initialize your parameter manually.<br>
<code>torch.Tensor</code> will use uninitialized memory and will thus contain random values and might also contain invalid values (NaN, Inf etc.).</p>