Sparsity constraint at the output layer of an Autoencoder, make always the same neuron to be active

username: MngFrc
cooked: <p>Hi everyone!</p>
<p>I am trying to implement an Autoencoder, whose layout is the following:</p>
<ul>
<li>Input layer : <strong>100</strong> neurons</li>
<li>Hidden layer 1 : <strong>40</strong> neurons</li>
<li>Hidden layer 2 : <strong>20</strong> neurons</li>
<li>Hidden layer 3 == <em>encoder output layer</em> : <strong>4</strong> neurons</li>
<li>Hidden layer 4 : <strong>20</strong> neurons</li>
<li>Hidden layer 5 : <strong>40</strong> neurons</li>
<li>Output layer : <strong>100</strong> neurons</li>
</ul>
<p>I use the MSE loss function; ReLu activation functions; Adam optimizer with weight decay.</p>
<p>I want (actually I need) to deploy also a sparsity constraint in the hidden layer 3. That means that I would like to have just one activated encoder output neuron for each input sample.<br>
In order to do so, I have implementing a KL additive los.<br>
This is the code:</p>
<pre><code class="lang-auto"># kl_divergence() takes two input parameters, rho and rho_hat. 
# rho is the sparsity parameter value (RHO) that we have initialized earlier.
# rho_hat is the output after going through the model layer-wise until hidden layer 3
# The first calculation for rho_hat happens layer-wise in sparse_loss(). Then we pass the values as inputs to kl_divergence()

def kl_divergence(rho, rho_hat):
    # rho_hat has dimension (batch_size)x(4), 4 being the number of output neurons

    # we count the number of activated neurons (remember: we have used ReLu)
    for i in range(rho_hat.shape[0]):
        for j in range(rho_hat.shape[1]):
            if rho_hat[i,j]&gt;0:
                rho_hat[i,j]= 1
            else:
                rho_hat[i,j]= 0
    # compute the average activation frequency of each neuron.
    rho_hat = ((rho_hat.sum(dim=0))/rho_hat.shape[0])
    # Now rho_hat has 4 values, each of which is the average activation frequency of the corresponding neuron
    # we want to avoid the logarithm being infinity
    for i in range(len(rho_hat)): # len(rho_hat)==4
        if rho_hat[i]==0:
            rho_hat[i]=0.000001
        elif rho_hat[i]==1:
            rho_hat[i]=0.999999

    rho = torch.tensor([rho] * len(rho_hat)).to(device)
    # we return the KL divergence between rho and rho_hat.
    return torch.sum(rho * torch.log(rho/rho_hat) + (1 - rho) * torch.log((1 - rho)/(1 - rho_hat))).item()
 
# define the sparse loss function
def sparse_loss(rho, x, feat_name):
    values = x  # input data
    loss = 0
    for i in range(len(model_children)):
        values = model_children[i](values)
        values = F.relu(values) 
        if i==2:
            loss += kl_divergence(rho, values)
            break
    return loss
</code></pre>
<p>So, for each batch, I count the number of times in which the neuron j is activated. Then I compute the average and the corresponding loss. RHO is the desired frequency of activation.<br>
An alternative could be to compute the loss along the entire dataset, then start the traininf phase, adding to the loss of each batch the computed sparsity penalty. I computed it, but the performance was, again, not satisfactory.</p>
<p>Implementing this, with RHO=0.25, it turns out that always <strong>the same</strong> neuron is active. 3 neurons out of 4 output always 0. This seems to me a strong limitation, obviously.</p>
<p>I would like to know if I am missing something, or if there is a logical bug in my code.</p>
<p><em>Edit</em>: Maybe also a NN’s parameters other than the default one could help, but I am not sufficiently prepared oh this topic. So if you have any suggestion is welcomed.</p>