RuntimeError: F.binary_cross_entropy all elements of input should be between 0 and 1

username: Hillary
cooked: <p>‘’'class Dir_VAE(nn.Module):<br>
def <strong>init</strong>(self):<br>
super(Dir_VAE, self).<strong>init</strong>()<br>
self.encoder = nn.Sequential(</p>
<pre><code>        nn.Conv2d(nc, ndf, 4,4, 0,bias=False),
        nn.LeakyReLU(0.2, inplace=True),
       
        nn.Conv2d(ndf, ndf * 2, 4,4, 0,bias=False),
        nn.BatchNorm2d(ndf * 2),
        nn.LeakyReLU(0.2, inplace=True),
       
        nn.Conv2d(ndf * 2, ndf * 4, 4, 4, 0,bias=False),
        nn.BatchNorm2d(ndf * 4),
        nn.LeakyReLU(0.2, inplace=True),
       
        nn.Conv2d(ndf * 4, 512, 4, 2,0,bias=False),
        nn.LeakyReLU(0.2, inplace=True),
      
    )

    self.decoder = nn.Sequential(
       
        nn.ConvTranspose2d(512, ngf * 4, 4, 2, 0, bias=False),
        nn.BatchNorm2d(ngf * 4),
        nn.ReLU(True),
        
        nn.ConvTranspose2d(ngf * 4, ngf * 2, 4, 4, 0, bias=False),
        nn.BatchNorm2d(ngf * 2),
        nn.ReLU(True),
       
        nn.ConvTranspose2d(ngf * 2, ngf * 2, 4, 4, 0, bias=False),
        nn.BatchNorm2d(ngf * 2),
        nn.ReLU(True),
        
        nn.ConvTranspose2d(ngf * 2, nc, 4, 4, 0, bias=False),
       
        nn.Sigmoid()
       
    )
    self.fc1 = nn.Linear(512, 256)
    self.fc21 = nn.Linear(256, 10)
    self.fc22 = nn.Linear(256, 10)

    self.fc3 = nn.Linear(10, 256)
    self.fc4 = nn.Linear(256, 512)

    self.lrelu = nn.LeakyReLU()
    self.relu = nn.ReLU()

    # Dir prior
    self.prior_mean, self.prior_var = map(nn.Parameter, prior(10, 0.3)) # 0.3 is a hyper param of Dirichlet distribution
    self.prior_logvar = nn.Parameter(self.prior_var.log())
    self.prior_mean.requires_grad = False
    self.prior_var.requires_grad = False
    self.prior_logvar.requires_grad = False


def encode(self, x):
    conv = self.encoder(x);
    print('Size', conv.shape)
    h1 = self.fc1(conv.view(-1, 512))
    return self.fc21(h1), self.fc22(h1)

def decode(self, gauss_z):
    dir_z = F.softmax(gauss_z,dim=1) 
    h3 = self.relu(self.fc3(dir_z))
    deconv_input = self.fc4(h3)
    print('Deconv ', deconv_input.shape)
    deconv_input = deconv_input.view(-1,512,1,1)
    return self.decoder(deconv_input)

def reparameterize(self, mu, logvar):
    std = torch.exp(0.5*logvar)
    eps = torch.randn_like(std)
    return mu + eps*std


def forward(self, x):
    mu, logvar = self.encode(x)
    gauss_z = self.reparameterize(mu, logvar) 
    # gause_z is a variable that follows a multivariate normal distribution
    # Inputting gause_z into softmax func yields a random variable that follows a Dirichlet distribution (Softmax func are used in decoder)
    dir_z = F.softmax(gauss_z,dim=1) # This variable follows a Dirichlet distribution
    return self.decode(gauss_z), mu, logvar, gauss_z, dir_z

# Reconstruction + KL divergence losses summed over all elements and batch
def loss_function(self, recon_x, x, mu, logvar, K):
    beta =0.9
    print('Recon ',recon_x.shape)
    print('Data ' ,x.shape)              
    BCE = F.binary_cross_entropy(recon_x.view(-1, 65536), x.view(-1, 65536), reduction='sum')
   
    
    KLD = -0.5 * torch.sum(1+logvar - mu**2 - torch.exp(logvar), axis=1)
    return BCE + beta*KLD```
</code></pre>
<pre><code class="lang-auto">RuntimeError                              Traceback (most recent call last)
&lt;ipython-input-8-ef9ef8984921&gt; in &lt;module&gt;
    267     # 学習(Train)
    268     for epoch in range(50):
--&gt; 269         train(epoch)
    270         test(epoch)
    271         with torch.no_grad():

&lt;ipython-input-8-ef9ef8984921&gt; in train(epoch)
    226         optimizer.zero_grad()
    227         recon_batch, mu, logvar, gauss_z, dir_z = model(data)
--&gt; 228         loss = model.loss_function(recon_batch, data, mu, logvar, 10)
    229         loss = loss.mean()
    230         loss.backward()

&lt;ipython-input-8-ef9ef8984921&gt; in loss_function(self, recon_x, x, mu, logvar, K)
    198         print('Recon ',recon_x.shape)
    199         print('Data ' ,x.shape)
--&gt; 200         BCE = F.binary_cross_entropy(recon_x.view(-1, 65536), x.view(-1, 65536), reduction='sum')
    201         # ディリクレ事前分布と変分事後分布とのKLを計算
    202         # Calculating KL with Dirichlet prior and variational posterior distributions

C:\Conda5\lib\site-packages\torch\nn\functional.py in binary_cross_entropy(input, target, weight, size_average, reduce, reduction)
   2760         weight = weight.expand(new_size)
   2761 
-&gt; 2762     return torch._C._nn.binary_cross_entropy(input, target, weight, reduction_enum)
   2763 
   2764 

RuntimeError: all elements of input should be between 0 and 1```'''</code></pre>

username: KFrank
cooked: <p>Hi Hillary!</p>
<aside class="quote no-group quote-modified" data-username="Hillary" data-post="1" data-topic="159779" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/h/3bc359/48.png" class="avatar"> Hillary:</div>
<blockquote>
<pre><code class="lang-auto">--&gt; 228         loss = model.loss_function(recon_batch, data, mu, logvar, 10)
...
--&gt; 200         BCE = F.binary_cross_entropy(recon_x.view(-1, 65536), x.view(-1, 65536), reduction='sum')
...
-&gt; 2762     return torch._C._nn.binary_cross_entropy(input, target, weight, reduction_enum)
...
RuntimeError: all elements of input should be between 0 and 1```'''
</code></pre>
</blockquote>
</aside>
<p>Look for <code>nan</code>s (or <code>inf</code>s) in <code>recon_batch</code>.  It’s not entirely clear how<br>
your code hooks together, and you don’t tell us what <code>model</code> is, but<br>
the error message is complaining about <code>input</code>, the first argument to<br>
<code>binary_cross_entropy()</code>, and <code>nan</code>s would be the most likely cause.</p>
<p>Given what I think your code is doing, you shouldn’t be getting values<br>
less than zero or greater than one in <code>recon_batch</code>, but you should<br>
check for values outside of <code>[0.0, 1.0]</code>, as well.</p>
<aside class="quote no-group">
<blockquote>
<pre><code class="lang-auto">        self.decoder = nn.Sequential(
           
            nn.ConvTranspose2d(512, ngf * 4, 4, 2, 0, bias=False),
            nn.BatchNorm2d(ngf * 4),
            nn.ReLU(True),
            
            nn.ConvTranspose2d(ngf * 4, ngf * 2, 4, 4, 0, bias=False),
            nn.BatchNorm2d(ngf * 2),
            nn.ReLU(True),
           
            nn.ConvTranspose2d(ngf * 2, ngf * 2, 4, 4, 0, bias=False),
            nn.BatchNorm2d(ngf * 2),
            nn.ReLU(True),
            
            nn.ConvTranspose2d(ngf * 2, nc, 4, 4, 0, bias=False),
           
            nn.Sigmoid()
           
        )
</code></pre>
</blockquote>
</aside>
<p>As an aside, for reasons of numerical stability, you should get rid<br>
of the final <code>Sigmoid</code> “layer” in <code>self.decoder</code> (so that it outputs<br>
<em>logits</em> rather than probabilities) and use<br>
<code>torch.nn.functional.binary_cross_entropy_with_logits()</code><br>
(instead of <code>binary_cross_entropy()</code>)  for that piece of your<br>
Kulback-Leibler-divergence <code>loss_function()</code>.</p>
<p>Best.</p>
<p>K. Frank</p>