Binary cross entropy unusual input and target VAE

alexobrads: <p>Hi Team,<br>
I am new to Pytorch so this might be a silly question but hopefully someone can help, I have tried my best over the past few days to debug/educate myself more to try and resolve my problem but have been unable to.</p>
<p>Set up:</p>
<ol>
<li>I am building a variational auto encoder</li>
<li>Data is just y values of some graph that has multiple Gaussians, there are 239 points.</li>
<li>I have 45k of these “y data”  arrays for training and 5k for testing</li>
<li>I am preprocessing my data by removing outliers and applying a whitening transformation which uses the mean and std per dimension calculated on all 50k sets of data</li>
</ol>
<p>VAE: I followed to blog post tutorial by Raviraja G<br>
<a href="https://graviraja.github.io/vanillavae/" class="onebox" target="_blank" rel="noopener nofollow ugc">https://graviraja.github.io/vanillavae/</a></p>
<p>I am using these parameters</p>
<pre><code class="lang-auto">BATCH_SIZE = 64     # number of data points in each batch
N_EPOCHS = 20      # times to run the model on complete data
INPUT_DIM = 239    # size of each input
HIDDEN_DIM = 50    # hidden dimension
LATENT_DIM = 5     # latent vector dimension
lr = 1e-7          # learning rate

#import for generalisation, so data is fed in randomly
SHUFFLE = True
</code></pre>
<p>and this is how I have set up the encoder, decoder and VAE</p>
<pre><code class="lang-auto">class Encoder(nn.Module):
    """ This the encoder part of VAE

    """
    def __init__(self, input_dim, hidden_dim, z_dim):

        super().__init__()

        self.linear = nn.Linear(input_dim, hidden_dim)
        self.mu = nn.Linear(hidden_dim, z_dim)
        self.var = nn.Linear(hidden_dim, z_dim)

    def forward(self, x):
        # x is of shape [batch_size, input_dim]

        hidden = F.relu(self.linear(x))
        # hidden is of shape [batch_size, hidden_dim]
        z_mu = self.mu(hidden)
        # z_mu is of shape [batch_size, latent_dim]
        z_var = self.var(hidden)
        # z_var is of shape [batch_size, latent_dim]

        return z_mu, z_var


class Decoder(nn.Module):
    """ This the decoder part of VAE
    """
    def __init__(self, z_dim, hidden_dim, output_dim):
        
        super().__init__()

        self.linear = nn.Linear(z_dim, hidden_dim)
        self.out = nn.Linear(hidden_dim, output_dim)

    def forward(self, x):
        # x is of shape [batch_size, latent_dim]

        hidden = F.relu(self.linear(x))
        # hidden is of shape [batch_size, hidden_dim]

        predicted = torch.sigmoid(self.out(hidden))
        # predicted is of shape [batch_size, output_dim]

        return predicted


class VAE(nn.Module):
    """ This is the VAE, which takes a encoder and decoder.

    """
    def __init__(self, enc, dec):
        super().__init__()

        self.enc = enc
        self.dec = dec

    def forward(self, x):
        # encode
        z_mu, z_var = self.enc(x)

        # sample from the distribution having latent parameters z_mu, z_var
        # reparameterize
        std = torch.exp(z_var / 2)
        eps = torch.randn_like(std)
        x_sample = eps.mul(std).add_(z_mu)

        # decode
        predicted = self.dec(x_sample)
        return predicted, z_mu, z_var

</code></pre>
<p>And I am running training and testing the following way.</p>
<pre><code class="lang-auto">def train(model):
    # set the train mode
    model.train()
    # loss of the epoch
    train_loss = 0
    for i, (x,n,f) in enumerate(train_iterator):
        # reshape the data into [batch_size, 784]
        x = x.view(-1, 239)
        x = x.to(device)

        # update the gradients to zero
        optimizer.zero_grad()

        # forward pass
        x_sample, z_mu, z_var = model(x.float())

        #reconstruction loss
        try:
            recon_loss = F.binary_cross_entropy(x_sample, x.float(), size_average=False)
        except:
            print(x)
            print(np.min(x.numpy()), np.max(x.numpy()), np.min(x.numpy()), np.max(x.numpy()))
            print(f[i])


        # kl divergence loss
        kl_loss = 0.5 * torch.sum(torch.exp(z_var) + z_mu**2 - 1.0 - z_var)
        # total loss
        loss = recon_loss + kl_loss

        # backward pass
        loss.backward()
        train_loss += loss.item()
        # update the weights
        optimizer.step()

    return train_loss



def test(model):
    # set the evaluation mode
    model.eval()
    # test loss for the data
    test_loss = 0
    # we don't need to track the gradients, since we are not updating the parameters during evaluation / testing
    with torch.no_grad():
        for i, (x,n,f) in enumerate(test_iterator):
            # reshape the data
            x = x.view(-1, 239)
            x = x.to(device)

            # forward pass
            x_sample, z_mu, z_var = model(x.float())
            #reconstruction loss
            try:
                recon_loss = F.binary_cross_entropy(x_sample, x.float(), size_average=False)
            except:
                print(x)
                print(np.min(x.numpy()), np.max(x.numpy()), np.min(x.numpy()), np.max(x.numpy()))
                print(f[i])
            # kl divergence loss
            kl_loss = 0.5 * torch.sum(torch.exp(z_var) + z_mu**2 - 1.0 - z_var)

            # total loss
            loss = recon_loss + kl_loss
            test_loss += loss.item()

    return test_loss


train_ll = []
test_ll = []

best_test_loss = float('inf')

for e in range(N_EPOCHS):

    train_loss = train(model)
    test_loss = test(model)

    train_loss /= len(train_dataset)
    test_loss /= len(test_dataset)

    print(f'Epoch {e}, Train Loss: {train_loss:.2f}, Test Loss: {test_loss:.2f}')
    train_ll.append(train_loss)
    test_ll.append(test_loss)


    if best_test_loss &gt; test_loss:
        best_test_loss = test_loss
        patience_counter = 1
    else:
        patience_counter += 1

    if patience_counter &gt; 3:
        break
</code></pre>
<p>THE PROBLEM:<br>
During the training/testing stage my F.Binary cross entropy fails as it says the values need to be between 0,1. This is where things get odd. before feeding my data into the VAE it appears to be scaled correctly. The target and input values for the BCE are usually between [-1, 3] and occasionally have a large or small value |3000| somewhere in that tensor.<br>
Occasionally it does train without failing, sometimes it fails. When it does not fail the loss does not really get better but the VAE kind of works?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/4/44a79d478c3dae64de6a95f69ce991b44ffd84eb.png" data-download-href="https://discuss.pytorch.org/uploads/default/44a79d478c3dae64de6a95f69ce991b44ffd84eb" title="Screen Shot 2020-06-07 at 7.08.30 pm"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44a79d478c3dae64de6a95f69ce991b44ffd84eb_2_321x375.png" alt="Screen Shot 2020-06-07 at 7.08.30 pm" data-base62-sha1="9NlzY0QODxOPozMU4WQe1mS8dxp" width="321" height="375" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44a79d478c3dae64de6a95f69ce991b44ffd84eb_2_321x375.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44a79d478c3dae64de6a95f69ce991b44ffd84eb_2_481x562.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/4/44a79d478c3dae64de6a95f69ce991b44ffd84eb_2_642x750.png 2x" data-dominant-color="B3B3B3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-06-07 at 7.08.30 pm</span><span class="informations">1072×1252 286 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>And here are the results, blue line is the original data and yellow is output of the same data through the VAE.</p>
<p>Questions:</p>
<ol>
<li>I am unsure if the way I have set up the VAE is the best way to do this problem, WHat are some other things I could change/test?</li>
<li>Why would the BCE fail only sometimes when nothing has been changed?</li>
<li>The error I get sometimes from BCE, What are some things I could do to check/debug what could be creating that.</li>
<li>How do I improve the VAE? should I bump up the epochs or something else? I am unsure how to interpret the Loss results.</li>
</ol>
<p>Any suggestions or thought would be greatly appreciated. thank you <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

ptrblck: <aside class="quote no-group" data-username="alexobrads" data-post="1" data-topic="84504">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3ab097/48.png" class="avatar"> alexobrads:</div>
<blockquote>
<p>The target and input values for the BCE are usually between [-1, 3] and occasionally have a large or small value |3000| somewhere in that tensor.</p>
</blockquote>
</aside>
<p>The raised error points most likely to the model output, which might not be in the required range of <code>[0, 1]</code>. Note that also the target should be in the same range, but it seems that you are using values outside of <code>[0, 1]</code>?</p>
<ol>
<li>
<p>Try to add print statements for <code>x_sample</code> and check the min and max values.</p>
</li>
<li>
<p>Unsure at the moment, but the error is usually raised, if the model output is out of the range.</p>
</li>
</ol>

alexobrads: <p>Thanks so much for getting back to me I really appreciate it!<br>
I did what you recommended and here are some of the results…<br>
<code>x_sample</code> is constantly between [0,1]. X doesn’t appear to be scaled as it should be. Here is an example of some of the values in the order, during training stage.<br>
<code>min(x_sample), max(x_sample), min(x), max(x)</code><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/2/420abbd69eee9cd567c21100cfac91104a5efe5b.png" data-download-href="https://discuss.pytorch.org/uploads/default/420abbd69eee9cd567c21100cfac91104a5efe5b" title="Screen Shot 2020-06-08 at 5.58.41 pm"><img src="https://discuss.pytorch.org/uploads/default/original/3X/4/2/420abbd69eee9cd567c21100cfac91104a5efe5b.png" alt="Screen Shot 2020-06-08 at 5.58.41 pm" data-base62-sha1="9qevCoIufgptQBQvOCbshORc6X9" width="517" height="292" data-dominant-color="494949"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2020-06-08 at 5.58.41 pm</span><span class="informations">842×476 26.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>If I understand correctly the X values should also be between [0,1] which suggests that my whitening method is incorrect. I am using the following formulas…</p>
<p><code>y* = (y - mean)/std</code></p>
<p>where mean is an array or means per input dimension from entire training set.<br>
and std is array of std per input dimension from entire training set.</p>
<p>If this is correct are there any reasons to why the data wouldn’t scale to the correct range [0, 1]?</p>
<p>The training now runs but the reconstruction loss is obviously useless currently due to the mis match between <code>x_sample </code> and x.</p>
<p>If you have any suggestion to correct my initial scaling or if something else appears wrong please let me know,<br>
again thank you very much for your time.</p>

ptrblck: <p>You are currently passing <code>x</code> as the target, so I’m not sure why you are normalizing it.<br>
Also, I don’t think that the current value range calculates what you expect given in the binary cross entropy formula.</p>
<aside class="quote no-group" data-username="alexobrads" data-post="4" data-topic="84504">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3ab097/48.png" class="avatar"> alexobrads:</div>
<blockquote>
<p>If this is correct are there any reasons to why the data wouldn’t scale to the correct range [0, 1]?</p>
</blockquote>
</aside>
<p>The posted formula will not normalize the tensor to the range <code>[0, 1]</code>, but will standardize it to a zero mean and unit variance.</p>
<aside class="quote no-group" data-username="alexobrads" data-post="4" data-topic="84504">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/3ab097/48.png" class="avatar"> alexobrads:</div>
<blockquote>
<p>If you have any suggestion to correct my initial scaling or if something else appears wrong please let me know,</p>
</blockquote>
</aside>
<p>I wouldn’t try to normalize the target, but leave it in the range <code>[0, 1]</code>.</p>
<p>Let me know, if I misunderstand your use case or code, please.</p>

alexobrads: <p>hmm ok I will try be a little clearer,<br>
To my understanding, (which might be wrong)<br>
X is the data before going into the encoder and <code>x_sample</code> is the data that has come out of the decoder.</p>
<p>Because I am using my own data I have done what I believe to be the correct preprocessing, which is to apply the whitening method described above which as you corrected me on, standardizes my data to 0, before feeding it into the VAE.</p>
<p>Are you suggesting that for BCE to work correctly, my target (The input to the VAE) has to be scaled to the range [0,1]? If this is correct, how would I do this? Should I scale it per piece of data that I have or should I take the min max out of my entire data set and scale by the global min max?</p>
<p>I am unclear on a) what should the target values look like. b) how they should be scaled and any preprocessing used on the target. c) that I am correct in thinking that  x (target) is input to VAE and x_sample (input) is the out put of the VAE,</p>
<p>Thanks again for your help</p>

ptrblck: <p>I would try to normalize the complete dataset to values in the range <code>[0, 1]</code> for the input and target.<br>
You might standardize the input e.g. in your <code>forward</code> method, but I’m not sure, if this would help or if it could even be harmful.</p>
<p>If you pass a target outside of <code>[0, 1]</code>, your loss might get negative, which seems weird to me (also I’m not sure what the target outside of <code>[0, 1]</code> would mean in theory for binary cross entropy).</p>

alexobrads: <p>Hey Patrick!</p>
<p>Your suggestions helped me figure out what was wrong, I had an extra sigmoid layer in the decoder and so the output was not being return to the correct domain as the input, hence BCE couldn’t calculate an accurate measure.</p>
<p>Thank you for your help <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>