Initalize the weights of nn.ConvTranspose2d

username: huxc_ustc
cooked: <p>how shoud I initalize  the weights of nn.ConvTranspose2d ? like nn.Conv2d? is this any special for Pytorch</p>
<p><em><strong>Add another question</strong></em>:Does pytorch require manual weight initialization or pytorch layers would initialize automatically?  means:if i do’t initialize the weight or bias ,it is all zero or random value ?</p>
<pre><code>for m in self.modules():
    if isinstance(m, nn.Conv2d):
        n = m.kernel_size[0] * m.kernel_size[1] * m.out_channels
        m.weight.data.normal_(0, math.sqrt(2. / n))
    elif isinstance(m, nn.BatchNorm2d):
        m.weight.data.fill_(1)
        m.bias.data.zero_()
</code></pre>
<p>infer :  the bias will automatically initialize with random value . is that right?</p>
<p>The work I want to do is to build a FCN based on a caffemodel FCN . now i want to initalize the network .as following<br>
my initalization:</p>
<pre><code>def weights_initG(m):
    for p in m.modules():
        if isinstance(p,nn.Conv2d):
            n = p.kernel_size[0] * p.kernel_size[1] * p.out_channels
            p.weight.data.normal_(0, math.sqrt(2. / n))
        elif isinstance(p,nn.BatchNorm2d):
            p.weight.data.normal_(1.0, 0.02)
            p.bias.data.fill_(0)
        elif isinstance(p,nn.ConvTranspose2d):
            n=p.kernel_size[1]
            factor = (n+1)//2
            if n%2 ==1:
                center = factor - 1
            else :
                center = factor -0.5
            og = np.ogrid[:n,:n]
            weights_np=(1-abs(og[0]-center)/factor)* (1-abs(og[1]-center)/ factor)
            p.weight.data.copy_(torch.from_numpy(weights_np))
</code></pre>
<p>Question one : how should I initalize the bias of Conv and deconv<br>
Question two :since Pytorch  image data is between[0,1],caffe image data is [0,255]. the weight initalization method  have any difference with Caffe?</p>

username: chenyuntc
cooked: <p>have a look at <a href="https://github.com/pytorch/examples/blob/master/dcgan/main.py" rel="nofollow noopener">example dcgan</a></p>
<pre><code class="lang-python">def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        m.weight.data.normal_(0.0, 0.02)
    elif classname.find('BatchNorm') != -1:
        m.weight.data.normal_(1.0, 0.02)
        m.bias.data.fill_(0)

netG.apply(weights_init)
</code></pre>
<p>it should work.</p>

username: david-leon
cooked: <p><a class="mention" href="/u/chenyuntc">@chenyuntc</a> Does pytorch require manual weight initialization or pytorch layers would initialize automatically? I noticed there is .reset_parameters() in the base _ConvNd class, but I didn’t see where this function is called.</p>

username: apaszke
cooked: <p>The parameters are initialized automatically. If you want to use a specific initialization strategy take a look at <a href="https://github.com/pytorch/pytorch/blob/master/torch/nn/init.py"><code>torch.nn.init</code></a>. I’ll need to add that to the docs.</p>

username: acgtyrant
cooked: <p><code>reset_parameters()</code> should be called in <code>__init__</code>.</p>

username: bille_du
cooked: <p>hello,hank you very much for your help, I ask you another question, for the above code, use the normal distribution to initialize the weights. If i want a normal distribution of variables in a certain range, what method can be done? Do I need to customize a distribution function?<br>
Or called Truncated Normal Distribution</p>

username: chenyuntc
cooked: <p>maybe just use <code>clamp_</code></p>
<pre><code class="lang-auto">m.weight.data.normal_(1.0, 0.02).clamp_(min=0,max=2)
</code></pre>

username: bille_du
cooked: <p>ok，thank you！by the way，if I want to clamp [-1, -0.1] and [0.1,1] 。How to operate？</p>

username: chenyuntc
cooked: <pre><code class="lang-auto">a.clamp_(min=-1,max=1)
a[a.abs()&lt;0.1]=t.sign(a[a.abs()&lt;0.1])*0.1
</code></pre>

username: bolt25
cooked: <p>Can anyone please explain how exactly this snippet works?<br>
I’m not able to get how the m.weight.data.normal_ line works<br>
and if I set  bias=False while defining my network is it necessary to define m.bias.data.fill_(0) explicitly again?</p>

username: ptrblck
cooked: <p><code>.normal_</code> fills the tensor inplace with values drawn from the normal distribution using the specified mean and standard deviation (<a href="https://pytorch.org/docs/stable/tensors.html#torch.Tensor.normal_" rel="nofollow noopener">docs</a>).</p>
<p>If you set <code>bias=False</code>, you don’t have to and in fact cannot call <code>m.bias.data</code>, since <code>bias</code> will be <code>None</code>.</p>
<p>Note, that I would recommend to wrap the parameter manipulations in a <code>torch.no_grad</code> block and avoid using the <code>.data</code> attribute directly:</p>
<pre><code class="lang-python">lin = nn.Linear(10, 10, bias=False)

with torch.no_grad():
    lin.weight.normal_(0.0, 1.0)
</code></pre>

username: bolt25
cooked: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> thank you! This helped me a lot<br>
Just one question what is the difference between using torch.no_grad() and using .data?</p>

username: ptrblck
cooked: <p><code>torch.no_grad</code> will disable gradient calculation, so that all operations in this block won’t be tracked by Autograd.<br>
Using the underlying <code>.data</code> attribute will most likely work in this case.<br>
However, I consider it dangerous, as it might silently introduce errors in the gradient calculations.</p>