Custom weight initialization

username: omrysendik
cooked: <p>Dear experienced ones,</p>
<p>What would be the right way to implement a custom weight initialization method?<br>
I believe I can’t directly add any method to torch.nn.init but wish to initialize my model’s weights with my own proprietary method.</p>
<p>Thanks!</p>

username: ptrblck
cooked: <p>Have a look at the other init implementations: <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/init.py">code</a>.<br>
It’s basically a vanilla Python function setting the values.<br>
Just create a new function with your proprietary method and call it with <code>model.apply(my_init)</code>.</p>

username: isalirezag
cooked: <p>Lets say this is my nn</p>
<pre><code class="lang-auto">
class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x
</code></pre>
<p>and I want my weights, be like this<br>
W = <code>torch.Tensor([[1 ,0, -1],[2, 0 ,-2], [1, 0 ,-1]])</code></p>
<p>Is there anyway to do that?</p>

username: rasbt
cooked: <blockquote>
<p>and I want my weights, be like this<br>
W = torch.Tensor([[1 ,0, -1],[2, 0 ,-2], [1, 0 ,-1]])</p>
<p>Is there anyway to do that?</p>
</blockquote>
<p>you don’t even have to implement a special PyTorch function for that. E.g. you can assign these values to</p>
<pre><code>with torch.no_grad():
    self.fc1.weight = ... 
</code></pre>
<p>where <code>...</code> would be replaced by a tensor (e.g. <code>W</code> in your case) or a function that creates a tensor with the right dimensions (assuming that in a real usecase, it’s probably to tedious to type out the whole tensor).</p>

username: isalirezag
cooked: <p>So for example for <code>self.conv1</code> can i do</p>
<pre><code class="lang-auto">with torch.no_grad():
    self.conv1.weight = ... 
</code></pre>
<p>???</p>

username: rasbt
cooked: <p>yes, that should work.</p>

username: isalirezag
cooked: <p>sorry for my stupid question,<br>
but i get this error:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3,3, 3)
        K = torch.Tensor([[1 ,0, -1],[2, 0 ,-2], [1, 0 ,-1]])
#        I think I should make the shape/size like this?
        K = torch.unsqueeze(torch.unsqueeze(K,0),0)
        with torch.no_grad():
            self.conv1.weight = K
        
    def forward(self, x):
        x = self.conv1(x)
        return x
    
    


net = Net()
net(rand(4,3,10,10))

TypeError: cannot assign 'torch.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)
</code></pre>

username: rasbt
cooked: <p>Ah sorry about that, I forgot to mention torch.nn.Parameter, which basically makes the weight recognizable as a parameter when you then do sth like</p>
<pre><code>optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)  
</code></pre>
<p>So, in your case, the following should fix it:</p>
<pre><code>    with torch.no_grad():
        self.conv1.weight = torch.nn.Parameter(K)
</code></pre>
<p>EDIT: I am actually not sure if it even requires no_grad, because this is still in the <code>__init__</code> part and not in the <code>forward</code> call.</p>

username: aerinykim
cooked: <p><a class="mention" href="/u/isalirezag">@isalirezag</a><br>
Did <a class="mention" href="/u/rasbt">@rasbt</a> 's suggestion work for you?</p>
<p>Because for me, it doesnt. Torch 0.4.</p>
<pre><code class="lang-auto">In [22]: emb.weight
Out[22]: 
Parameter containing:
tensor([[-0.5733,  0.8611,  0.0699],
        [ 1.7558,  0.7413, -0.4029],
        [-0.7413, -1.0902,  0.0550],
        [-0.9363, -0.6691, -0.4870],
        [ 0.0080, -0.4598,  0.9987]])

In [23]: K = torch.Tensor([[1 ,0, -1],[2, 0 ,-2], [1, 0 ,-1],[3,4,5],[6,7,8]])

In [24]: K
Out[24]: 
tensor([[ 1.,  0., -1.],
        [ 2.,  0., -2.],
        [ 1.,  0., -1.],
        [ 3.,  4.,  5.],
        [ 6.,  7.,  8.]])

In [25]: emb.weight = emb.weight+ nn.Parameter(K)
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
&lt;ipython-input-25-9f658db7039f&gt; in &lt;module&gt;()
----&gt; 1 emb.weight = emb.weight+ nn.Parameter(K)

~/anaconda3/envs/san/lib/python3.6/site-packages/torch/nn/modules/module.py in __setattr__(self, name, value)
    549                 raise TypeError("cannot assign '{}' as parameter '{}' "
    550                                 "(torch.nn.Parameter or None expected)"
--&gt; 551                                 .format(torch.typename(value), name))
    552             self.register_parameter(name, value)
    553         else:

TypeError: cannot assign 'torch.FloatTensor' as parameter 'weight' (torch.nn.Parameter or None expected)
</code></pre>
<p>How do I add arbitrary tensor to embedding’s weight?</p>

username: aerinykim
cooked: <p><code>emb.weight.data = emb.weight.data+ nn.Parameter(K)</code></p>
<p><code>weight.data</code> worked.</p>

username: vmirly1
cooked: <p>But i think since you have used <code>.data</code>, it is unneccesary to convert that K to <code>Parameter</code>. You can remove the <code>with torch.no_grad()</code> block and remove the nn.Parameter.</p>

username: vmirly1
cooked: <p>So, the revised code example will be as follows:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn

class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        self.conv1 = nn.Conv2d(3,3, 3)
        K = torch.Tensor([[1 ,0, -1],[2, 0 ,-2], [1, 0 ,-1]])
#        I think I should make the shape/size like this?
        K = torch.unsqueeze(torch.unsqueeze(K,0),0)
        #with torch.no_grad():
        self.conv1.weight.data = self.conv1.weight.data + K

    def forward(self, x):
        x = self.conv1(x)
        return x

net = Net()
net(torch.randn(4,3,10,10))
</code></pre>

username: aerinykim
cooked: <p>What if you want to optimize the weight?</p>

username: vmirly1
cooked: <p>The weight will be optimized. It’s just that the initial values have changed as the question is how to use the custom initialization.</p>

username: rasbt
cooked: <p>Vahid is right that in the case of his example</p>
<pre><code>self.conv1.weight.data = self.conv1.weight.data + K
</code></pre>
<p>this will work because “weight” is already a parameter, and you are just modifying its value. But if you want to assign a completely new tensor to “weight” you would need wrap <code>Parameter</code> around that to get correct behavior.</p>

username: Haze
cooked: <p>But there is still a problem what if we do use nn.sequential to bulid our model,how colud we  initial our model with this method?Just like this:</p>
<pre><code class="lang-auto">class Net(nn.Module):
    def __init__(self):                                             
        super(Net,self).__init__()
        #卷积层
        self.conv1 =torch.nn.Sequential(
            torch.nn.Conv1d(4,64,kernel_size=3,stride=1,padding=1),

            torch.nn.ReLU(),
            torch.nn.Conv1d(64,128,kernel_size=3,stride=1,padding=1),
            torch.nn.ReLU(),
            torch.nn.MaxPool1d(stride=2,kernel_size=2)
        )
        #全连接层
        self.dense=torch.nn.Sequential(
            torch.nn.Linear(640,320),
            torch.nn.ReLU(),
            torch.nn.Dropout(p=0.5),
            torch.nn.Linear(320,10)
        )
    def forward(self,x):
        x=self.conv1(x)
        x=x.view(-1,640)
        x=self.dense(x)
        return x  
</code></pre>

username: vmirly1
cooked: <p>You can still do that. I created an instance of the class <code>Net</code>:</p>
<pre><code class="lang-python">&gt;&gt;&gt; net = Net()
&gt;&gt;&gt; net
Net(
  (conv1): Sequential(
    (0): Conv1d(4, 64, kernel_size=(3,), stride=(1,), padding=(1,))
    (1): ReLU()
    (2): Conv1d(64, 128, kernel_size=(3,), stride=(1,), padding=(1,))
    (3): ReLU()
    (4): MaxPool1d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
  )
  (dense): Sequential(
    (0): Linear(in_features=640, out_features=320, bias=True)
    (1): ReLU()
    (2): Dropout(p=0.5)
    (3): Linear(in_features=320, out_features=10, bias=True)
  )
)
</code></pre>
<p>So, we can access the layers of this object by their names and their index:</p>
<pre><code class="lang-python">&gt;&gt;&gt; net.conv1[0]
Conv1d(4, 64, kernel_size=(3,), stride=(1,), padding=(1,))
</code></pre>
<p>So, we can assign the weights of each layer similarly:</p>
<pre><code class="lang-python">net.conv1[0].weight.data = net.conv1[0].weight.data + K
</code></pre>

username: Haze
cooked: <p>It is so great，thank you so much!!</p>
<p><a href="https://maas.mail.163.com/dashi-web-extend/html/proSignature.html?iconUrl=http%3A%2F%2Fmail-online.nosdn.127.net%2F6588e96a909b4a466e02b0546381a37d.jpg&amp;name=litianfudt&amp;uid=example%40163.com&amp;ftlId=3&amp;items=%5B%22%E9%82%AE%E7%AE%B1%EF%BC%9Alitianfudt%40163.com%22%5D" rel="nofollow noopener"><br>
</a><br>
<div class="lightbox-wrapper"><a class="lightbox" href="http://mail-online.nosdn.127.net/6588e96a909b4a466e02b0546381a37d.jpg" title="6588e96a909b4a466e02b0546381a37d.jpg" rel="nofollow noopener"><img width="38" height="38" src="http://mail-online.nosdn.127.net/6588e96a909b4a466e02b0546381a37d.jpg"><div class="meta">
<svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use xlink:href="#far-image"></use></svg><span class="filename">6588e96a909b4a466e02b0546381a37d.jpg</span><span class="informations">750×750</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use xlink:href="#discourse-expand"></use></svg>
</div></a></div><br>
litianfudt</p>
<p>邮箱：litianfudt@163.com</p>
<p>Signature is customized by <a href="https://mail.163.com/dashi/dlpro.html?from=mail88" rel="nofollow noopener">Netease Mail Master</a></p>

username: vmirly1
cooked: <p>Sure, I am happy to help!</p>