How do i init parameters partially?

Yeo_Wei_jie: <p>I want to apply transfer learning by loading the state dict of one model to another. However, in the final layer which is a linear layer of 2D dim, i would like to randomly initalise it partially. Meaning say the linear layer is of size [3,256], how do i apply <code>torch.nn.init.xavier_uniform_</code> to only 20% of the parameters in the linear layer?</p>

KFrank: <p>Hi Wei Jie!</p>
<aside class="quote no-group" data-username="Yeo_Wei_jie" data-post="1" data-topic="149877" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yeo_wei_jie/48/48918_2.png" class="avatar"> Yeo_Wei_jie:</div>
<blockquote>
<p>how do i apply <code>torch.nn.init.xavier_uniform_</code> to only 20% of the parameters in the linear layer?</p>
</blockquote>
</aside>
<p>You could use a number of approaches.</p>
<p>I would probably use <code>multinomial()</code> to choose which elements of<br>
the <code>Linear</code> to partially initialize.  Then, really just for convenience, I<br>
would apply <code>xavier_uniform_()</code> to a temporary tensor of the same<br>
shape as the <code>Linear.weight</code> in question.  Lastly, I would index into<br>
the <code>weight</code> tensor to update the desired elements inplace.</p>
<p>For example:</p>
<pre><code class="lang-plaintext">&gt;&gt;&gt; import torch
&gt;&gt;&gt; print (torch.__version__)
1.10.2
&gt;&gt;&gt;
&gt;&gt;&gt; _ = torch.manual_seed (2022)
&gt;&gt;&gt;
&gt;&gt;&gt; pretrained_linear = torch.nn.Linear (3, 5)
&gt;&gt;&gt; xavier_weight = torch.nn.init.xavier_uniform_ (torch.empty_like (pretrained_linear.weight))
&gt;&gt;&gt;
&gt;&gt;&gt; pretrained_linear.weight
Parameter containing:
tensor([[-0.1203,  0.4872,  0.2988],
        [-0.1372, -0.5471, -0.1624],
        [ 0.3387,  0.3245, -0.0412],
        [ 0.1535,  0.1951, -0.3115],
        [ 0.3467,  0.2717,  0.2861]], requires_grad=True)
&gt;&gt;&gt; xavier_weight
tensor([[ 5.6547e-01, -4.9935e-01,  6.1908e-01],
        [-1.2431e-01, -1.7949e-01, -6.1666e-01],
        [-8.6022e-01,  7.8006e-01, -4.8605e-01],
        [-3.6499e-01,  3.0906e-01, -5.1421e-04],
        [ 7.1781e-01, -3.7528e-01,  1.2801e-01]])
&gt;&gt;&gt;
&gt;&gt;&gt; mask = torch.multinomial (torch.tensor ([0.8, 0.2]), pretrained_linear.weight.nelement(), replacement = True).reshape_as (pretrained_linear.weight)
&gt;&gt;&gt; mask
tensor([[1, 1, 0],
        [0, 0, 0],
        [1, 0, 0],
        [1, 1, 0],
        [0, 0, 0]])
&gt;&gt;&gt;
&gt;&gt;&gt; inds = torch.nonzero (mask)
&gt;&gt;&gt; inds
tensor([[0, 0],
        [0, 1],
        [2, 0],
        [3, 0],
        [3, 1]])
&gt;&gt;&gt;
&gt;&gt;&gt; with torch.no_grad():
...     pretrained_linear.weight[inds[:, 0], inds[:, 1]] = xavier_weight[inds[:, 0], inds[:, 1]]
...
&gt;&gt;&gt; pretrained_linear.weight
Parameter containing:
tensor([[ 0.5655, -0.4994,  0.2988],
        [-0.1372, -0.5471, -0.1624],
        [-0.8602,  0.3245, -0.0412],
        [-0.3650,  0.3091, -0.3115],
        [ 0.3467,  0.2717,  0.2861]], requires_grad=True)
</code></pre>
<p>Best.</p>
<p>K. Frank</p>

Yeo_Wei_jie: <aside class="quote no-group" data-username="KFrank" data-post="2" data-topic="149877">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/k/ecb155/48.png" class="avatar"> KFrank:</div>
<blockquote>
<p><code>torch.multinomial (torch.tensor ([0.8, 0.2]), pretrained_linear.weight.nelement(), replacement = True).reshape_as (pretrained_linear.weight)</code></p>
</blockquote>
</aside>
<p>Thanks frank! that work out well.</p>