How to create a sparse autoencoder neural network with pytorch

username: mingqiang_chen
cooked: <p>how to create a sparse autoEncoder neural network with pytorch,tanks!</p>

username: smth
cooked: <p>in a sparse autoencoder, you just have an L1 sparsitiy penalty on the intermediate activations.</p>
<p>You can create a <code>L1Penalty</code> autograd function that achieves this.</p>
<pre><code class="lang-python">import torch
from torch.autograd import Function

class L1Penalty(Function):

    @staticmethod
    def forward(ctx, input, l1weight):
        ctx.save_for_backward(input)
        ctx.l1weight = l1weight
        return input

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_variables
        grad_input = input.clone().sign().mul(self.l1weight)
        grad_input += grad_output
        return grad_input
</code></pre>
<p>Then you can use it like this:</p>
<pre><code class="lang-auto">import torch.nn.functional as F

class SparseAutoEncoder(torch.nn.Module):
  def __init__(self, feature_size, hidden_size, l1weight):
    super(SparseAutoEncoder, self).__init__()
    self.lin_encoder = nn.Linear(feature_size, hidden_size)
    self.lin_decoder = nn.Linear(hidden_size, feature_size)
    self.feature_size = feature_size
    self.hidden_size = hidden_size
    self.l1weight = l1weight
    
  def forward(input):
    # encoder
    x = input.view(-1, self.feature_size)
    x = self.lin_encoder(x)
    x = F.relu(x)

    # sparsity penalty
    x = L1Penalty.apply(x, self.l1weight)
    
    # decoder
    x = self.lin_decoder(x)
    x = F.sigmoid(x)
    return x.view_as(input)

</code></pre>
<p>I didn’t test the code for exact correctness, but hopefully you get an idea.</p>
<p>References:</p>
<ol>
<li><a href="https://github.com/Kaixhin/Autoencoders/blob/master/models/SparseAE.lua">https://github.com/Kaixhin/Autoencoders/blob/master/models/SparseAE.lua</a></li>
<li><a href="https://github.com/torch/nn/blob/master/L1Penalty.lua">https://github.com/torch/nn/blob/master/L1Penalty.lua</a></li>
</ol>

username: Ben
cooked: <p><a class="mention" href="/u/smth">@smth</a></p>
<p>Just can’t connect the code with the document.<br>
<a href="http://deeplearning.stanford.edu/wiki/index.php/Autoencoders_and_Sparsity" class="onebox" target="_blank" rel="nofollow noopener">http://deeplearning.stanford.edu/wiki/index.php/Autoencoders_and_Sparsity</a></p>
<p>Can you show me some more details? Where is the parameter of sparsity? What is l1weight? Is it the parameter of sparsity, e.g. 5%? What is the loss function?<br>
Is there any completed code?</p>
<p>Thanks in advance!</p>

username: vaxin
cooked: <p>Why put L1Penalty into a Layer? Why dont add it to the loss function?<br>
In another words, L1Penalty in just one activation layer will be automatically added into the final loss function by pytorch itself?</p>

username: 1112
cooked: <p>what is the difference with adding l1 or KL-loss to final loss function ?</p>

username: Shisho_Sama
cooked: <p>This code doesnt run in Pytorch 1.1.0! I keep getting the backward() needs to return two values not 1!<br>
Edit :<br>
You need to return None for any arguments that you do not need the gradients. so the L1Penalty would be :</p>
<pre><code class="lang-auto">import torch
from torch.autograd import Function

class L1Penalty(Function):

    @staticmethod
    def forward(ctx, input, l1weight):
        ctx.save_for_backward(input)
        ctx.l1weight = l1weight
        return input

    @staticmethod
    def backward(ctx, grad_output):
        input, = ctx.saved_variables
        grad_input = input.clone().sign().mul(self.l1weight)
        grad_input += grad_output
        return grad_input, None
</code></pre>

username: Shisho_Sama
cooked: <p>Did you find your answer?</p>