How to initialize weight with arbitrary tensor

MORI: <p>Hi,</p>
<p>I would like to initialize weights and bias with arbitrary values, not with uniforms or constant number.</p>
<p>class Model(nn.Module):<br>
def <strong>init</strong>(self,init_param):<br>
super(Model, self).<strong>init</strong>()<br>
self.conv1 = nn.Conv2d(512, 512, 3)<br>
self.conv1.weight = initial_param[‘weight’]<br>
self.conv1.bias = initial_param[‘bias’]</p>
<pre><code>def forward(self, x):
   return F.relu(self.conv2(x))
</code></pre>
<p>model = Model(initial_param)</p>
<p>initial_param[‘weight’] and initial_param['bias] are torch.FloatTensor of size 512x512x3x3 and 512 respectively.</p>
<p>I got following error<br>
TypeError: cannot assign ‘torch.FloatTensor’ as child module ‘conv1’ (torch.nn.Module or None expected)</p>
<p>How to assign arbitrary values to parameters?</p>
<p>Now, my purpose is convert torch(lua) model to pytorch model.( torch.legacy.nn.Sequential to pytorch model)<br>
I’m trying to extract weight and bias from legacy model and assign to pytorch model since there is no way to do it automatically.</p>
<p>Thanks.</p>

mbp28: <p>Hey,</p>
<p>I think something like this could work for you:</p>
<pre><code>self.conv1.weight = torch.nn.Parameter(initial_param['weight'])
self.conv1.bias = torch.nn.Parameter(initial_param['bias'])
</code></pre>
<p>For example, the following code runs without error:</p>
<pre><code>import torch
from torch.autograd import Variable
import torch.nn as nn
import torch.nn.functional as F

class Kernel_Emb(nn.Module):

    def __init__(self,D_in,H,D_out):
        super(Kernel_Emb, self).__init__()
        self.linear1 = nn.Linear(D_in,H)
        self.linear2 = nn.Linear(H,D_out)
        self.linear1.weight = torch.nn.Parameter(torch.zeros(D_in,H))
        self.linear1.bias = torch.nn.Parameter(torch.ones(H))
    
    
    def forward(self, x):
        h_relu = self.linear1(x).clamp(min=0)
        y_pred = self.linear2(h_relu)
        return y_pred

net = Kernel_Emb(embd_dim,H1,kernel_dim)
print(net.linear1.bias)  ###this prints ones.</code></pre>

MORI: <p>It worked! Thanks a lot!</p>

Came: <p>Based on the discussion here,<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/2159" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <a href="https://github.com/yunjey" rel="nofollow noopener">
<img src="https://avatars1.githubusercontent.com/u/15663219?v=2&amp;s=96" class="thumbnail onebox-avatar" width="60" height="60">
</a>

<h4><a href="https://github.com/pytorch/pytorch/issues/2159" target="_blank" rel="nofollow noopener">Issue: Efficient forward pass in nn.Linear</a></h4>

<div class="date" style="margin-top:10px;">
	<div class="user" style="margin-top:10px;">
	opened by <a href="https://github.com/yunjey" target="_blank" rel="nofollow noopener">yunjey</a>
	on <a href="https://github.com/pytorch/pytorch/issues/2159" target="_blank" rel="nofollow noopener">2017-07-20</a>
	</div>
	<div class="user">
	closed by <a href="https://github.com/soumith" target="_blank" rel="nofollow noopener">soumith</a>
	on <a href="https://github.com/pytorch/pytorch/issues/2159" target="_blank" rel="nofollow noopener">2017-07-20</a>
	</div>
</div>

<pre class="content" style="white-space: pre-wrap;">Hi,
Maybe we can improve the operation speed in nn.Linear.
nn.Linear creates a weight matrix of shape (out_features, in_features) as you can see...</pre>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
I think the shape pf weight matrix in linear layer should be reverse.</p>
<p>change:<br>
self.linear1.weight = torch.nn.Parameter(torch.zeros(D_in,H))<br>
to<br>
self.linear1.weight = torch.nn.Parameter(torch.zeros(H,D_in))</p>

yimeng_min: <aside class="quote no-group" data-username="Came" data-post="4" data-topic="3432">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/came/48/9868_2.png" class="avatar"> Came:</div>
<blockquote>
<p>I think the shape pf weight matrix in linear layer should be reverse.</p>
</blockquote>
</aside>
<p>you are right, the parameters should be in the reverse order</p>

Ting_liu: <p>Thank you very much!</p>