What is the default initial weights for pytorch-geometric SAGEconv layer and other convolution layers? and how to initialize them using Xavier?

username: Mary_H
cooked: <p>Hello,<br>
what is the default initial weights for pytorch-geometric SAGEconv layer and other convolution layers? and how to initialize them using Xavier?</p>
<p>I need guidance in how to apply Xavier initialization in graph neural network, I am using pytorch geometry</p>
<p>I tried this, but it didn’t work,</p>
<blockquote>
<p>def initialize_weights(self):<br>
for m in self.modules():<br>
if isinstance(m,SAGEConv):<br>
nn.init.xavier_normal_(m.weight)<br>
if m.bias is not None:<br>
nn.init.constant_(m.bias, 0)</p>
</blockquote>
<p><strong>it neither goes to the first if statement, and gives error “AttributeError(f”‘{type(self).<strong>name</strong>}’ object has no attribute ‘{name}’“) AttributeError: ‘MeanAggregation’ object has no attribute ‘bias’”</strong></p>

username: J_Johnson
cooked: <p>The default PyTorch conv class uses Kaiming uniform for weights and a standard uniform distribution for biases.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/a/aaa909b91f7d32da586993a79ac2e1e5cd654ac2.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/aaa909b91f7d32da586993a79ac2e1e5cd654ac2" title="IMG_20231221_220720"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aaa909b91f7d32da586993a79ac2e1e5cd654ac2_2_544x500.jpeg" alt="IMG_20231221_220720" data-base62-sha1="olJkATSF7QCJrG18hTpB7BEBuGC" width="544" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aaa909b91f7d32da586993a79ac2e1e5cd654ac2_2_544x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aaa909b91f7d32da586993a79ac2e1e5cd654ac2_2_816x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/a/a/aaa909b91f7d32da586993a79ac2e1e5cd654ac2_2_1088x1000.jpeg 2x" data-dominant-color="EDEDED"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20231221_220720</span><span class="informations">1240×1138 157 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m not sure for SAGEConv, but for regular PyTorch layers, you can just define a custom conv class, and reference the parent accordingly, then redefine the <code>reset_parameters()</code> function. Or just modify your code to something like this:</p>
<pre><code class="lang-auto">def initialize_weights(self):
    for m in self.modules():
        if isinstance(m,SAGEConv):
            nn.init.xavier_normal_(m.weight.data)
</code></pre>
<p>Note the <code>.data</code>.</p>

username: Mary_H
cooked: <p>when I tried using m.weight.data, I still get this error,<br>
‘SAGEConv’ object has no attribute ‘weight’<br>
do you have any idea ?</p>

username: J_Johnson
cooked: <p>Had a look at the SAGEConv module docs and it appears they just create a separate linear class, also defined within that library:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/7/1715bc755da5e0479984128d526dae5640e2fec8.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/1715bc755da5e0479984128d526dae5640e2fec8" title="IMG_20231226_072027"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/7/1715bc755da5e0479984128d526dae5640e2fec8_2_269x500.jpeg" alt="IMG_20231226_072027" data-base62-sha1="3idyblYuOtSyzJgqVHz4eN3CKb6" width="269" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/7/1715bc755da5e0479984128d526dae5640e2fec8_2_269x500.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/1/7/1715bc755da5e0479984128d526dae5640e2fec8_2_403x750.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/7/1715bc755da5e0479984128d526dae5640e2fec8_2_538x1000.jpeg 2x" data-dominant-color="E7E7E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20231226_072027</span><span class="informations">1240×2304 134 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
So calling on the SAGEConv class won’t target the weights. Instead, you may need to change <code>SAGEConv</code> to <code>Linear</code>. But given it’s not a torch.nn.Linear, that might not work, too.</p>
<p>They also have a specific reset_parameters method:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/0/e00264cd105cc9b9bc9c5aafb5449550b16c77ff.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/e00264cd105cc9b9bc9c5aafb5449550b16c77ff" title="IMG_20231226_072345"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/0/e00264cd105cc9b9bc9c5aafb5449550b16c77ff_2_690x106.jpeg" alt="IMG_20231226_072345" data-base62-sha1="vXG556Ve35ATZ81MBonHDrnW75J" width="690" height="106" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/0/e00264cd105cc9b9bc9c5aafb5449550b16c77ff_2_690x106.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/e/0/e00264cd105cc9b9bc9c5aafb5449550b16c77ff_2_1035x159.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/0/e00264cd105cc9b9bc9c5aafb5449550b16c77ff_2_1380x212.jpeg 2x" data-dominant-color="E8E7E8"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20231226_072345</span><span class="informations">2467×382 85.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Digging further, we can find the specific reset_weight definition:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/e/6e57d8abced0c4fc0c9ae7719d923a97bdd6dd10.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/6e57d8abced0c4fc0c9ae7719d923a97bdd6dd10" title="IMG_20231226_072636"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e57d8abced0c4fc0c9ae7719d923a97bdd6dd10_2_690x317.jpeg" alt="IMG_20231226_072636" data-base62-sha1="fK8JmMl9hykz8KGZusqZ5aj7CBa" width="690" height="317" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e57d8abced0c4fc0c9ae7719d923a97bdd6dd10_2_690x317.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e57d8abced0c4fc0c9ae7719d923a97bdd6dd10_2_1035x475.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/e/6e57d8abced0c4fc0c9ae7719d923a97bdd6dd10_2_1380x634.jpeg 2x" data-dominant-color="E9E9EA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20231226_072636</span><span class="informations">1920×883 80.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And it appears that takes an initialization argument.</p>
<p>However, I do not see Xavier listed as an initialization option.</p>
<p>Perhaps you could check on the Slack linked on their docs homepage:<br>
<a href="https://pytorch-geometric.readthedocs.io/en/latest/index.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch-geometric.readthedocs.io/en/latest/index.html</a></p>
<p>Seems to me the issue is not related to PyTorch.</p>

username: Mary_H
cooked: <p>I tried to print out the weights and I found that it is initialised by default with uniform Kaiming, and the weights are in conv1.lin_l.weight and conv1.lin_r.weight. for each SAGEconv layer. Reaching both of these weights, I could initialise them using Xavier.<br>
thanks.</p>