Implementing k-sparse autoencoder on FastText embedding, the output is strange

username: kjang0517
cooked: <p>Hi, I’m implementing k-Sparse Autoencoders (A. Makhzani et al., 2013). With the implementation I’m trying to sparse-code my pre-trained word embeddings. Below is the algorithm explained in the paper. In my understanding, it’s basically overcomplete autoencoder which has constraint of only selecting top-k activations from the hidden layer.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/0739e3e4ad57954da2457e19459a2783e788b823.png" data-download-href="https://discuss.pytorch.org/uploads/default/0739e3e4ad57954da2457e19459a2783e788b823" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0739e3e4ad57954da2457e19459a2783e788b823_2_571x500.png" alt="image" data-base62-sha1="11Vn20VT21YcacMJcDzodgfsBWP" width="571" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/0739e3e4ad57954da2457e19459a2783e788b823_2_571x500.png, https://discuss.pytorch.org/uploads/default/original/2X/0/0739e3e4ad57954da2457e19459a2783e788b823.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/0/0739e3e4ad57954da2457e19459a2783e788b823.png 2x" data-dominant-color="EFEFEF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">748×654 179 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What I’m struggling is that the most of the resulting sparse embeddings (top-k activations for each embedding rows) are all 0s, even though the loss is properly decreasing. Of course the goal of sparse coding is preserving only the information (activation) that matters, but the problem is there’s no information left!</p>
<p>I’m attaching the code segments for the advice. Here’s the full <a href="https://www.dropbox.com/s/rpby3rfu9ixg0b4/1_K_Sparse_Autoencoder.py?dl=0" rel="noopener nofollow ugc">code</a>, and the <a href="https://www.dropbox.com/s/cej539g36nio6b2/fasttext_wiki_size_300_min_count_5_iter_50_negative_20_ngrams_0.hdf5?dl=0" rel="noopener nofollow ugc">data</a>. The data is <code>.hdf5</code> file which has fields <code>['words', 'vectors']</code></p>
<p><strong># EmbeddingDataset</strong><br>
This consists of <code>vectors</code> (np array, ‘f4’) and words (np array, str, not used in training)</p>
<pre data-code-wrap="python"><code class="lang-python">class EmbeddingDataset(data.Dataset):
    def __init__(self, vectors, words):
        self.vectors = vectors
        self.words = words
    
    def __len__(self):
        return len(self.words)
    
    def __getitem__(self, index):
        vectors = self.vectors[index]
        words = self.words[index]
        return vectors, words
</code></pre>
<p><strong># KSparseAutoencoder</strong></p>
<ul>
<li>Hidden size = 1000, k = e.g. 15 (making 300-dimensional embedding into sparse 1000-dimensional ones, which has at most <code>k</code> non-zero dimensions)
<ul>
<li>starting with k=15 might produce many dead-activations even before finding gradients, so k gradually decreases from 100 to 15 (the authors’ suggestion)</li>
</ul>
</li>
<li>In the encode stage, I use cap-ReLU (min = 0, max = 1) to make the resulting embedding non-negative, regularized.</li>
<li>I select top-k activations in the <code>forward</code> method, making others zero. Then I proceed to decode stage only with the surviving values. <strong>Please check the code for any errors.</strong></li>
</ul>
<pre data-code-wrap="python"><code class="lang-python">class KSparseAutencoder(nn.Module):
    def __init__(self, D_in_out, H, k, total_epoch, use_activation=True):
        """
        In the constructor we instantiate two nn.Linear modules and assign them as
        member variables.
        """
        super().__init__()
        self.encoder = nn.Linear(D_in_out, H)
        self.decoder = nn.Linear(H, D_in_out)
        self.H = H
        self.k = k
        self.use_activation = use_activation
        k_grace_period = total_epoch // 2
        k_start = max(k*2, 100)
        logging.info(f"k = {k_start}~{k}")
        self.k_list = [math.floor(k) for k in np.linspace(k_start, k, num=k_grace_period)] + [k for _ in range(total_epoch-k_grace_period)]
        
    def forward(self, x, epoch, is_final=False):
        """
        In the forward function we accept a Tensor of input data and we must return
        a Tensor of output data. We can use Modules defined in the constructor as
        well as arbitrary operators on Tensors.
        """
        # Encode
        encoded = self.encoder(x)
        if self.use_activation:
            encoded = encoded.clamp(min=0, max=1) # cap-ReLU

        # Select k
        for e in encoded:
            if is_final:
                k = int(self.k * 1.5)
            else:
                k = self._get_k(epoch)
            _, indices_to_erase = torch.topk(e, (self.H - k), largest=False)
            e[indices_to_erase] = 0.0

        # Decode
        decoded = self.decoder(encoded)
        return encoded, decoded
    
    def _get_k(self, epoch):
        return math.ceil(self.k_list[epoch])
</code></pre>
<p><strong>Training</strong></p>
<p>UPDATE: I suspect calculating the loss like this <code>loss = criterion(vectors, decoded)</code> is causing the problem. Is it a correct way to evaluation a batch of embeddings?</p>
<pre data-code-wrap="python"><code class="lang-python">with torch.cuda.device(params['gpu_idx']):
        model = KSparseAutencoder(D_in_out, H, k, num_epochs, use_activation).to(device)
        learning_rate = 0.1
        optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
        criterion = nn.MSELoss()
        for epoch in range(num_epochs):
            # Training
            epoch_loss = 0.0
            max_sparsity = 0.0
            for vectors, _ in data_generator:
                # Get batch
                vectors = Variable(vectors, requires_grad=False).type(dtype)

                # denoising
                if denoising and noise_level &gt; 0.0:
                    noise = get_noise_features(vectors.shape[0], vectors.shape[1], noise_level)
                    noise = Variable(torch.from_numpy(noise.astype('f4')), requires_grad=False).type(dtype)
                    vectors += noise
                
                # Move data to GPU
                vectors = vectors.to(device)

                # Forward
                encoded, decoded = model(vectors, epoch)
                max_sparsity = max(max_sparsity, check_sparsity(encoded))

                # Loss
                loss = criterion(vectors, decoded)
                epoch_loss += criterion(vectors, decoded)

                # Zero gradients, perform a backward pass, and update the weights.
                optimizer.zero_grad()
                loss.backward()
                optimizer.step()

            avg_epoch_loss = epoch_loss / len(words)
            N = 20
            print(vectors[0][:N])
            print(decoded[0][:N])
            print(f"After epoch {epoch}, Avg. Loss = {avg_epoch_loss:.6f}, Sparsity = {max_sparsity}")
</code></pre>

username: 111324
cooked: <p>If you manully place some values of a tensor to zeros, it will not compute the gradients properly.</p>
<p>I implent a <code>k sparse filter</code>, you can test it out.</p>
<pre><code class="lang-python">import torch


class KSparse(torch.nn.Module):
    def __init__(self, k):

        super(KSparse, self).__init__()
        self.k = k

    def forward(self, x):

        _, indices = torch.topk(x, self.k)
        mask = torch.zeros(x.size()).cuda()
        mask.scatter_(2, indices, 1)
        return torch.mul(x, mask)


test = torch.tensor([[[1., 2., 4.], [2., 3., 1.]]]).cuda()
k_sparse = KSparse(1)
print(k_sparse(test))
</code></pre>

username: kjang0517
cooked: <p>much appreciated! Thanks!</p>