Weight initialisation on custom Model

Prakhar_Sharma: <p>I constructed a custom model which looks like this (open the figure in a new tab). It is a combination of multiple <code>class</code>es.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/6/8606431940fb02c433974ab606c562cd69be70ed.png" data-download-href="https://discuss.pytorch.org/uploads/default/8606431940fb02c433974ab606c562cd69be70ed" title="DGM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/8606431940fb02c433974ab606c562cd69be70ed_2_433x499.png" alt="DGM" data-base62-sha1="j7DoE82aQA6s4MXyzDgbhSNBF9P" width="433" height="499" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/8606431940fb02c433974ab606c562cd69be70ed_2_433x499.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/8606431940fb02c433974ab606c562cd69be70ed_2_649x748.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/8/6/8606431940fb02c433974ab606c562cd69be70ed_2_866x998.png 2x" data-dominant-color="F2F2F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">DGM</span><span class="informations">1508×1740 170 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>The figure shows a single custom hidden layer (everything in between the input and output).</p>
<p>Finally I created a object using</p>
<pre data-code-wrap="python"><code class="lang-python">DGM_model = DGMArch(len(input_tensor), len(output_tensor))
</code></pre>
<p>I wanted to implement Glorot normal initialisation (<code>torch.nn.init.xavier_uniform_()</code> ). But the <code>DGM_model.weight</code> is not available.</p>
<p>I can implement this by iterating over each parameter using <a href="https://stackoverflow.com/a/57116138/14598633" rel="noopener nofollow ugc">this article</a>.</p>
<p>How do I implement this inside the <code>__init__()</code> function of the <code>DGMArch</code> class? Here is my weight initialisation code for sequential models.</p>
<pre data-code-wrap="python"><code class="lang-python">for i in range(len(layers)-1):
            
            # weights from a normal distribution with 
            # Recommended gain value for tanh = 5/3?
            nn.init.xavier_normal_(self.linears[i].weight.data, gain=1.0)
            
            # set biases to zero
            nn.init.zeros_(self.linears[i].bias.data)
</code></pre>

ptrblck: <aside class="quote no-group" data-username="Prakhar_Sharma" data-post="1" data-topic="159320">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/prakhar_sharma/48/47007_2.png" class="avatar"> Prakhar_Sharma:</div>
<blockquote>
<p>I wanted to implement Glorot normal initialisation (<code>torch.nn.init.xavier_uniform_()</code> ). But the <code>DGM_model.weight</code> is not available.</p>
</blockquote>
</aside>
<p>You would have to access the right layers inside the model as e.g.:</p>
<pre data-code-wrap="python"><code class="lang-python">torch.nn.init.xavier_uniform_(DGM_model.layer.weight)
</code></pre>
<p>where <code>DGM_model.layer</code> is accessing the <code>nn.Module</code> which you can then use to access the internal parameters of this layer via <code>.weight</code> and <code>.bias</code>.</p>

Prakhar_Sharma: <p>I wanted to implement the function within the <code>DGM-model</code> class. This is how I got this done.</p>
<pre><code class="lang-python">for l in range(nr_layers): # creating the DGM arch by appending all DGM layers
    self.layers.append(DGMLayer(d_in, layer_size, activation_fn, weight_norm)) # creates one DGM cell
        
    # difference between nn.Sequential and nn.Module: https://discuss.pytorch.org/t/when-should-i-use-nn-modulelist-and-when-should-i-use-nn-sequential/5463/9
    # basically, nn.Sequential is more tailored.
        self.layers = nn.Sequential(*self.layers) # for the last layer  # pass layers to nn.Sequential
        
    # Initialising custom weights
    if isinstance(self, nn.Linear): # for each weights matrix
        torch.nn.init.xavier_uniform_(self.weight)
</code></pre>