LIBTORCH/C++ How to initialize weights (Xavier) in a Sequential module with Apply function?

cheggars: <p>I tried implementing the approach described here: <a href="https://stackoverflow.com/questions/49433936/how-to-initialize-weights-in-pytorch" rel="nofollow noopener">https://stackoverflow.com/questions/49433936/how-to-initialize-weights-in-pytorch</a></p>
<p>i.e. creating a function that could be passed to Apply on all modules part of the sequential net. Here is the code:</p>
<p>=============================================</p>
<p>void  Init_Weights(torch::nn::Module&amp; m)<br>
{<br>
if (typeid(m) == typeid(torch::nn::Linear))<br>
{		<br>
torch::nn::init::xavier_normal_(m-&gt;weight);<br>
torch::nn::init::constant_(m-&gt;bias, 0.01);<br>
}<br>
}</p>
<p>int main() {</p>
<pre><code>torch::nn::Sequential XORModel(
	torch::nn::Linear(2, 3),
	torch::nn::Functional(torch::tanh),
	torch::nn::Linear(3, 1),
	torch::nn::Functional(torch::sigmoid));


XORModel-&gt;apply(Init_Weights);
</code></pre>
<p>}</p>
<p>=================================================<br>
The problem is that the code will not compile. Torch requires that definition of Init_Weights should have ‘torch::nn::Module&amp; m’ as input. In this case ‘m-&gt;weight’ could not be resolved as type Module does not have ‘weight’<br>
If I change definition of Init_Weights so that its input is of ‘torch::nn::Linear&amp; m’ than Init_weights could not be passed to Apply.</p>
<p>Is there a way to initialize (Xavier normal) all weight in Linear modules part of Sequential module?</p>

cheggars: <p>Just to update - I found a solution. See code below:</p>
<p>void  Init_Weights(torch::nn::Module&amp; m)<br>
{</p>
<pre><code>if ((typeid(m) == typeid(torch::nn::LinearImpl)) || (typeid(m) == typeid(torch::nn::Linear))) {
	auto p = m.named_parameters(false);
	auto w = p.find("weight");
	auto b = p.find("bias");

	if (w != nullptr) torch::nn::init::xavier_uniform_(*w);
	if (b != nullptr) torch::nn::init::constant_(*b, 0.01);
}
</code></pre>
<p>}</p>

cheggars: <p>Even better solution in the manual - <a href="https://pytorch.org/cppdocs/api/classtorch_1_1nn_1_1_module.html#exhale-class-classtorch-1-1nn-1-1-module" rel="nofollow noopener">https://pytorch.org/cppdocs/api/classtorch_1_1nn_1_1_module.html#exhale-class-classtorch-1-1nn-1-1-module</a></p>
<p><em>template</em> &lt; <em>typename</em> ModuleType, typename = <a href="https://pytorch.org/cppdocs/api/typedef_namespacetorch_1_1detail_1ae60d551fe64129deca3ea8d2d6d8fe06.html#namespacetorch_1_1detail_1ae60d551fe64129deca3ea8d2d6d8fe06" rel="nofollow noopener">torch::detail::disable_if_module_holder_t</a>&gt;<br>
ModuleType * <code>as</code> ()</p>
<p>Attempts to cast this <code>Module</code> to the given <code>ModuleType</code> .</p>
<p>This method is useful when calling <code>apply()</code> .</p>
<p>void initialize_weights(nn::Module&amp; module) { torch::NoGradGuard no_grad; if (auto* linear = module.as<a>nn::Linear</a>()) { linear-&gt;weight.normal_(0.0, 0.02); } } MyModule module; module.apply(initialize_weights);</p>

Aaditya_Chandrasekha: <p>I am not sure I follow this solution. Could you kindly elaborate on how you achieved the initialization ?</p>
<p>Thanks!</p>

quevivasbien: <p>In case anyone runs across this particular question again, the following should work as a simple solution:</p>
<pre><code class="lang-auto">void xavier_init(torch::nn::Module&amp; module) {
	torch::NoGradGuard noGrad;
	if (auto* linear = module.as&lt;torch::nn::Linear&gt;()) {
		torch::nn::init::xavier_normal_(linear-&gt;weight);
		torch::nn::init::constant_(linear-&gt;bias, 0.01);
	}
}
</code></pre>
<p>Then for any Linear module or module with Linear submodules, you can just initialize via <code>module-&gt;apply(xavier_init)</code>. I think this is basically what <a class="mention" href="/u/cheggars">@cheggars</a> was suggesting with their second response.</p>

JamesDickens: <p><a href="https://discuss.pytorch.org/u/JamesDickens"><img src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jamesdickens/90/45317_2.png" alt="" width="45" height="45"></a></p>
<p><a href="https://discuss.pytorch.org/u/JamesDickens">JamesDickens</a><a href="https://discuss.pytorch.org/u/JamesDickens">James McCulloch Dickens</a></p>
<p><a href="https://discuss.pytorch.org/t/libtorch-c-how-to-initialize-weights-xavier-in-a-sequential-module-with-apply-function/44920/6">1m</a></p>
<p>What does the torch::NoGradGuard noGrad; line do?</p>

AidanShipperley: <p>It’s the same thing as Pytorch’s <code>with torch.no_grad():</code>, it’s just the C++ equivalent. They talk more about layer initialization and grad guards in this thread: <a href="https://discuss.pytorch.org/t/initialization-in-place-and-tensor-data/74166" class="inline-onebox">Initialization in-place and `tensor.data`</a>, and it’s also explained in the documentation here: <a href="https://pytorch.org/docs/stable/notes/autograd.html#no-grad-mode" class="inline-onebox" rel="noopener nofollow ugc">Autograd mechanics — PyTorch 2.1 documentation</a></p>
<blockquote>
<p>The implementations in <a href="https://pytorch.org/docs/stable/nn.init.html#nn-init-doc" rel="noopener nofollow ugc">torch.nn.init</a> also rely on no-grad mode when initializing the parameters as to avoid autograd tracking when updating the intialized parameters in-place.</p>
</blockquote>