Weight initilzation

Hamid: <p>How I can initialize variables, say Xavier initialization?</p>

Atcold: <p>Hi <a class="mention" href="/u/hamid">@Hamid</a>,</p>
<p>I think you can extract the network’s parameters <code>params = list(net.parameters())</code> and then apply the initialisation you may like.<br>
If you need to apply the initialisation to a specific <code>module</code>, say <code>conv1</code>, you can extract the specific parameters with <code>conv1Params = list(net.conv1.parameters())</code>. You will have the kernels in <code>conv1Params[0]</code> and the bias terms in <code>conv1Params[1]</code>.</p>

fmassa: <p>Another possibility, which is present in the examples, can be found <a href="https://github.com/pytorch/examples/blob/master/dcgan/main.py#L90-L96">here</a>. This function specifies how the weights should be handled, and the weights are modified in <a href="https://github.com/pytorch/examples/blob/master/dcgan/main.py#L131">this line</a>.</p>

michael_k: <p><a href="https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py#L112-L118" rel="nofollow noopener">Another initialization example</a> from PyTorch Vision resnet implementation.</p>

Kalamaya: <p><a class="mention" href="/u/atcold">@Atcold</a> Can you give an example of what you mean? Thanks!</p>

Atcold: <p><a class="mention" href="/u/kalamaya">@Kalamaya</a>, I believe <a class="mention" href="/u/fmassa">@fmassa</a> is a cleaner solution.<br>
You traverse all <code>Module</code>s, and, upon <code>__class__.__name__</code> matching, you initialise the parameters with what you prefer.<br>
My method presupposes that you know the order of the <code>Module</code>s in the <code>_modules</code> <code>OrderedDict()</code>.</p>
<p>Does it make any sense what I am saying? If it does not, I can try better and with an example.</p>

Kalamaya: <p><a class="mention" href="/u/atcold">@Atcold</a> Yes, an example will help here since I am still traversing unfamiliar territory… <img src="//discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"> thank you for your help in advanced! Much obliged. If it helps, I am basically trying to initialize my conv and fully connected layers that I have. (I’d like to do Xavier, or fan_in fan_out etc). Thanks!</p>

Atcold: <p>You first define your name check function, which applies selectively the initialisation.</p>
<pre><code class="lang-python">def weights_init(m):
    classname = m.__class__.__name__
    if classname.find('Conv') != -1:
        xavier(m.weight.data)
        xavier(m.bias.data)
</code></pre>
<p>Then you traverse the whole set of <code>Modules</code>.</p>
<pre><code class="lang-python">net = Net() # generate an instance network from the Net class
net.apply(weights_init) # apply weight init
</code></pre>
<p>And this is it. You just need to define the <code>xavier()</code> function.</p>

apaszke: <p>A less Lua way of doing that would be to check if some module is an instance of a class. This is the recommended way:</p>
<pre><code class="lang-python">def weights_init(m):
    if isinstance(m, nn.Conv2d):
        xavier(m.weight.data)
        xavier(m.bias.data)
</code></pre>
<p><a class="mention" href="/u/atcold">@Atcold</a> another thing, accessing members prefixed with underscore is not recommended. They’re internal and subject to change without notice. If you want to get a iterator over modules use <code>.modules()</code> (searches recusrively) or <code>.children()</code> (only one level).</p>

Hamid: <p>Thanks guys.<br>
I am trying to apply weight initialization to a fully connected network (nn.Linear). I need, however, fan_out and fan_in of this layer. By fan_out and fan_in I mean number of output neurons and input neurons, respectively. How can I access them?</p>

Atcold: <p><a class="mention" href="/u/hamid">@Hamid</a>, you can check the size of the <code>weight</code> matrix.</p>
<pre><code class="lang-python">size = m.weight.size() # returns a tuple
fan_out = size[0] # number of rows
fan_in = size[1] # number of columns
</code></pre>
<p><a class="mention" href="/u/apaszke">@apaszke</a>, thanks for the heads-up! I’m still new to the Python world…</p>
<p>Edit: apply <a class="mention" href="/u/apaszke">@apaszke</a> fix.</p>

apaszke: <p>A small note - <code>.size()</code> is also defined on Variables, so no need to unpack the data. <code>m.weight.size()</code> will work too.</p>

Atcold: <p><a class="mention" href="/u/hamid">@Hamid</a>, are you trying to ask something? I am not sure I understand.<br>
Could you also please format your code with three backticks and the word <code>python</code>, so that I can read what you posted?</p>

Hamid: <p><a class="mention" href="/u/atcold">@Atcold</a>,  by checking if isinstance(m, nn.Linear) it would apply to linear module, correct?<br>
If I call the weight initialization, it would be applied to all layers?<br>
I have a residual module each having 2 linear layers. Then several of these modules.</p>
<p>my code:</p>
<pre><code>def weight_init(m): 
	if isinstance(m, nn.Linear):
		size = m.weight.size()
		fan_out = size[0] # number of rows
		fan_in = size[1] # number of columns
		variance = np.sqrt(2.0/(fan_in + fan_out))
		m.weight.data.normal_(0.0, variance)


class Residual(nn.Module):
	def __init__(self,dropout, shape, negative_slope, BNflag = False):
		super(Residual, self).__init__()
		self.dropout = dropout
		self.linear1 = nn.Linear(shape[0],shape[1])
		self.linear2 = nn.Linear(shape[1],shape[0])
		self.dropout = nn.Dropout(self.dropout)
		self.BNflag = BNflag
		self.batch_normlization = nn.BatchNorm1d(shape[0])
		self.leakyRelu = nn.LeakyReLU(negative_slope = negative_slope , inplace=False)

	def forward(self, X):
		x = X
		if self.BNFlag:
			x = self.batch_normlization(x)
		x = self.leakyRelu(x)
		x = self.dropout(x)
		x = self.linear1(x)
		if self.BNFlag:
			x = self.batch_normlization(x)
		x = self.leakyRelu(x)
		x = self.dropout(x)
		x = self.linear2(x)
		x = torch.add(x,X)
		return x
		
		
class FullyCN(nn.Module):
	def __init__(self, args):
		super(FullyCN, self).__init__()
		self.numlayers = arg.sm-num-hidden-layers
		self.learning-rate= args.sm-learning-rate
		self.dropout = arg.sm-dropout-prob
		self.BNflag = args.sm-bn
		self.shape = [args.sm-input-size,args.sm-num-hidden-units]		
		self.res =  Residual(self.dropout,self.shape,args.sm_act_param,self.self.BNflag)
		self.res(weight_init)
		self.res-outpus = []

	def forward(self,X):
		self.res-outpus.append(self.res(X))
		for i in range(self.numlayers):
			self.res-outpus.append(self.res(self.res-outpus[-1]))
		return self.res-outpus[-1]</code></pre>

Hamid: <p>sorry about confusion</p>

Atcold: <aside class="quote no-group" data-username="Hamid" data-post="15" data-topic="157" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/5fc32e/48.png" class="avatar"> Hamid:</div>
<blockquote>
<p><a class="mention" href="/u/atcold">@Atcold</a>,  by checking if isinstance(m, nn.Linear) it would apply to linear module, correct?</p>
</blockquote>
</aside>
<p>Correct.</p>
<aside class="quote no-group">
<blockquote>
<p>If I call the weight initialization, it would be applied to all layers?</p>
</blockquote>
</aside>
<p>Yup. All <strong>linear</strong> layers.</p>
<aside class="quote no-group">
<blockquote>
<p>I have a residual module each having 2 linear layers. Then several of these modules.</p>
</blockquote>
</aside>
<p>Sure, it will apply the normalisation to each <code>Module</code> that belongs to the class <code>nn.Linear</code>.</p>

Hamid: <aside class="quote no-group" data-username="Hamid" data-post="15" data-topic="157">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/5fc32e/48.png" class="avatar"> Hamid:</div>
<blockquote>
<p>self.res(weight_init)</p>
</blockquote>
</aside>
<p>But I have called weight_init once for the class while I call linear layers in a for loop (i.e., there are multiple sets of variables).</p>

Atcold: <pre><code class="lang-python">net = Residual() # generate an instance network from the Net class
net.apply(weights_init) # apply weight init
</code></pre>
<p>I’m not too sure what you’re doing with <code>FullyCN</code>…</p>

apaszke: <p>The <code>apply</code> function will search recursively for all the modules inside your network, and will call the function on each of them. So all <code>Linear</code> layers you have in your model will be initialized using this one call.</p>

Hamid: <p><a class="mention" href="/u/atcold">@Atcold</a>, in FullyCN I use several residual modules. In another piece of code, I pass data to FullyCN which returns corresponding output via its forward function.</p>