Randomly set the weights of part of model layers for every time training

shirui-japina: <ul>
<li>
<p>Is there some good way to set(re-set ) part layers’ weights randomly for every time train the model? (<strong>‘every time’</strong> means when every time <strong>train one time epoch</strong>.)</p>
</li>
<li>
<p>In addition, What is the <strong>usual range</strong> of weights?</p>
</li>
</ul>

Prashant_Kalikotay: <p>What do you mean by setting and resetting weights at different layers??</p>
<p>You can look at different initialisation like <a href="https://jamesmccaffrey.wordpress.com/2017/06/21/neural-network-glorot-initialization/" rel="nofollow noopener">Xavier glorot initialisation</a> and some other.<br>
Initalisation of weights depend on what data you are working with.</p>

shirui-japina: <p>I’m sorry, I did not express clearly.<br>
I mean, for <em><strong>specific layers of a model,</strong></em> set(re-set) layers’ weights <em><strong>randomly</strong></em> for every time training. Is there some good way to do this?</p>

Prashant_Kalikotay: <p>I did not get your question well. But to my understanding.<br>
If building your own function like convolution or something == True<br>
you can initialise your weights as <a href="https://pytorch.org/docs/stable/_modules/torch/nn/parameter.html" rel="nofollow noopener">torch.nn.Parameter</a>.</p>
<p>else<br>
you can use smthing like this:</p>
<pre><code class="lang-auto">def init_params(m):
  if type(m)==nn.Linear or type(m)==nn.Conv2d:
    m.weight.data=torch.randn(m.weight.size())*.01#Random weight initialisation
    m.bias.data=torch.zeros(m.bias.size())

</code></pre>

shirui-japina: <p>Appreciate your timely help.<img src="https://discuss.pytorch.org/images/emoji/apple/blush.png?v=9" title=":blush:" class="emoji" alt=":blush:"><br>
What does the <strong>‘m’</strong> in the function  mean as a parameter?<br>
I guess it’s each layer in the model.</p>
<ul>
<li>If so, which function in PyTorch I should use to get the layers?</li>
<li>If not, would you please tell me what it means?</li>
</ul>

Prashant_Kalikotay: <p>m in my previous reply  is a layer.</p>
<p>a simple example of layer or say a for example a 2D-Convolutional layer maybe:</p>
<pre><code class="lang-auto">import torch.nn as nn
layer=nn.Conv2d(in_channels=1,out_channels=32,kernel_size=3,stride=1)

</code></pre>
<p><a href="https://pytorch.org/docs/stable/nn.html" rel="nofollow noopener">torch.nn.Conv2D</a></p>
<p>I think that is too basic a question. Never mind but. You can check out Deep learning with pytorch Courses online.<a href="https://www.udacity.com/course/deep-learning-pytorch--ud188" rel="nofollow noopener">One such course</a></p>

shirui-japina: <p>I’m sorry, again, I didn’t express clearly.<img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
I mean, if the ‘m’ is a layer, then how to <em><strong>get and re-set the layer’s weights</strong></em> during <em><strong>the period of training</strong></em> the model? is there some good way to do this?</p>

Prashant_Kalikotay: <pre><code class="lang-auto">def init_params(m):
  if type(m)==nn.Linear or type(m)==nn.Conv2d:
    m.weight.data=torch.randn(m.weight.size())*.01#Random weight initialisation
    m.bias.data=torch.zeros(m.bias.size())

#for setting the weights you can use:
Model.apply(init_params) #Model here is the model that you have created

</code></pre>

shirui-japina: <p>It’s the first time I see the use of function, <code>model.apply()</code>. It’s very fantastic.<br>
Thank you very much for your help <img src="https://discuss.pytorch.org/images/emoji/apple/grinning.png?v=9" title=":grinning:" class="emoji" alt=":grinning:"></p>