Standard L2 norm weight regularisation

banikr: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e565009ed068f744db5117648e2aaa39ac1e8a9.png" data-download-href="https://discuss.pytorch.org/uploads/default/1e565009ed068f744db5117648e2aaa39ac1e8a9" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/e/1e565009ed068f744db5117648e2aaa39ac1e8a9_2_690x284.png" alt="image" data-base62-sha1="4knfnL4Fs8osydV8o72plRA9Obv" width="690" height="284" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/e/1e565009ed068f744db5117648e2aaa39ac1e8a9_2_690x284.png, https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e565009ed068f744db5117648e2aaa39ac1e8a9.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/1/e/1e565009ed068f744db5117648e2aaa39ac1e8a9.png 2x" data-dominant-color="F0EEEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">813×335 44.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>How can I perform this L2 norm weight regularisation in the following VAE network.</p>
<pre><code class="lang-auto">class VAE_msi(nn.Module):
    def __init__(self, inputsize, latent_dims):
        super(VAE_msi, self).__init__()
        self.N = torch.distributions.Normal(0, 1)
        self.layer_en1 = self.layer_(inputsize, 512)
        self.layer_en_xtra = self.layer_(512, 512)
        self.layer_en2 = self.layer_(512, latent_dims)
        self.layer_en3 = self.layer_(512, latent_dims)
        self.layer_de1 = self.layer_(latent_dims, 512)
        self.layer_de_xtra = self.layer_(512, 512)
        self.layer_de2 = self.layer_(512, inputsize, relu=False)
        self.N.loc = self.N.loc.to(device)  # hack to get sampling on the GPU
        self.N.scale = self.N.scale.to(device)
        self.dropout = nn.Dropout(0.5)

    #         self.kl = 0
    def layer_(self, inputsize, outputsize, relu=True):
        if relu:
            layer = nn.Sequential(
                nn.Linear(in_features=inputsize, out_features=outputsize),
                nn.BatchNorm1d(num_features=outputsize),
                nn.ReLU()
            )
        else:
            layer = nn.Sequential(
                nn.Linear(in_features=inputsize, out_features=outputsize),
                nn.BatchNorm1d(num_features=outputsize)
            )
        return layer

    def forward(self, spec):
        # ------- encoder -------#
        spec = self.dropout(spec)
        spec = self.layer_en1(spec)
        spec = self.layer_en_xtra(spec)
        mu = self.layer_en2(spec)
        sigma = self.layer_en3(spec)
        std = torch.exp(0.5 * sigma)
        eps = torch.randn_like(std)

        z = mu + eps * std
        # ------- decoder -------#
        z = self.dropout(z)
        z = self.layer_de1(z)
        z = self.layer_de_xtra(z)
        spec_hat = torch.sigmoid(self.layer_de2(z))
        return spec_hat, mu, sigma

</code></pre>
<p>How do I include this weight regularisation in the cost function as:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/e/7/e78f1981d997548c6476ebee4ece81074d535f39.png" alt="image" data-base62-sha1="x2sSSPSYFxsJ7042ssc8vNYjwRr" width="369" height="115"></p>