Generator model giving nan during .eval()

sahni: <p>Hi,</p>
<p>I am using the following generator model for a project, which is similar to DCGAN tutorial. The only difference is that I have added a couple of Residual Blocks in the beginning. In train mode, everything works fine and proper results are generated. However, if I set the model to eval mode using .eval(), then the model generates NaN output.</p>
<p>I have narrowed it down to an issue in the residual block, but I am not sure why NaN is being generated. I am suspecting that the issue might be due to the Instance Norm layer. Could someone clarify why this could be happening?</p>
<p>Note: Input to the generator is a Bx40x1x1 tensor, which consists of 1s ans 0s.</p>
<pre><code class="lang-auto">class ResidualBlock(nn.Module):
    """Residual Block with instance normalization."""
    def __init__(self, dim_in, dim_out):
        super(ResidualBlock, self).__init__()
        self.main = nn.Sequential(
            nn.Conv2d(dim_in, dim_out, kernel_size=3, stride=1, padding=1, bias=False),
            nn.InstanceNorm2d(dim_out, affine=True, track_running_stats=True),
            nn.ReLU(inplace=True),
            nn.Conv2d(dim_out, dim_out, kernel_size=3, stride=1, padding=1, bias=False),
            nn.InstanceNorm2d(dim_out, affine=True, track_running_stats=True))

    def forward(self, x):
        return x + self.main(x)

class Generator(nn.Module):
    def __init__(self, in_dim=40, conv_dim=64, out_dim=3):
        super(Generator, self).__init__()
        self.feat_extractor = nn.Sequential(
            # Residual Blocks
            ResidualBlock(in_dim, in_dim),
            ResidualBlock(in_dim, in_dim),
            ResidualBlock(in_dim, in_dim),
            ResidualBlock(in_dim, in_dim),
            ResidualBlock(in_dim, in_dim),
            ResidualBlock(in_dim, in_dim),
            # input is Z, going into a convolution
            nn.ConvTranspose2d(in_channels=in_dim, out_channels=conv_dim*8,
                                kernel_size=4, stride=1, padding=0, bias=False),
            nn.BatchNorm2d(conv_dim * 8),
            nn.ReLU(True),
            # state size. (conv_dim*8) x 4 x 4
            nn.ConvTranspose2d(in_channels=conv_dim*8, out_channels=conv_dim*4,
                                kernel_size=4, stride=2, padding=1, bias=False),
            nn.BatchNorm2d(conv_dim * 4),
            nn.ReLU(True),
            # state size. (conv_dim*4) x 8 x 8
            nn.ConvTranspose2d(in_channels=conv_dim*4, out_channels=conv_dim*2,
                                kernel_size=4, stride=2, padding=1, bias=False),
            nn.BatchNorm2d(conv_dim * 2),
            nn.ReLU(True),
            # state size. (conv_dim*2) x 16 x 16
            nn.ConvTranspose2d(in_channels=conv_dim*2, out_channels=conv_dim,
                                kernel_size=4, stride=2, padding=1, bias=False),
            nn.BatchNorm2d(conv_dim),
            nn.ReLU(True),
        )
        # state size. (conv_dim) x 32 x 32
        self.img_gen = nn.Sequential(
            nn.ConvTranspose2d(in_channels=conv_dim, out_channels=out_dim,
                                kernel_size=4, stride=2, padding=1, bias=False),
            nn.Tanh()
        )
        # state size. (out_dim) x 64 x 64

    def forward(self, input):
        return self.img_gen(self.feat_extractor(input))
</code></pre>

Nikronic: <p>Hi,</p>
<p>I have tested your model separately and together with <code>ResidulBlock</code> and it never gererates <code>NaN</code> values.</p>
<p>Here is the code I have tested:</p>
<pre><code class="lang-python">
model = Generator()
model.eval()
x = torch.randn(1, 40, 1, 1)
x[x&gt;0] = 0
x[x&lt;0] = 1
o = model(x)
print(o.size())  #torch.Size([1, 3, 64, 64])
</code></pre>

sahni: <p>Hi,</p>
<p>I think it is not generating the NaN values for you because when you are testing, the model is not trained yet. And the Instance Norm layer is also not trained.</p>
<p>I verified this on my system. Without loading the weights for the trained model, .eval mode works fine. But after loading the weights, it generates NaN.</p>

Nikronic: <p>I thought the problem pertain the implementation. Have you tried a different system? like google colab?</p>
<p>Actually, in a model of mine, the loss values explodes in my local environemnt(old system) but on colab model trains very well.</p>

sahni: <p>I think the problem pertains to some missing understanding of the instance norm layer. Since the code works perfectly in train mode, that is if I do evaluation without calling .eval, I don’t think it should be an environment issue.</p>
<p>I am wondering if their is some issue in instance norm for handling input which is predominantly zero during eval state.</p>

Idan_Azuri: <p>Have a look here:<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/5406" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/5406" target="_blank" rel="nofollow noopener">BatchNorm behaves different in train() and eval()</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2018-02-26" data-time="01:38:08" data-timezone="UTC">01:38AM - 26 Feb 18 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2018-02-26" data-time="04:03:16" data-timezone="UTC">04:03AM - 26 Feb 18 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/yuandong-tian" target="_blank" rel="nofollow noopener">
          <img alt="yuandong-tian" src="https://avatars3.githubusercontent.com/u/2973937?v=4" class="onebox-avatar-inline" width="20" height="20">
          yuandong-tian
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">import torch
import torch.nn as nn
from torch.autograd import Variable
print(torch.__version__)
def get_data(cuda=False):
 if cuda:
 return torch.cuda.FloatTensor(32, 1, 19, 19)
 else:
 return torch.FloatTensor(32, 1, 19,...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
In short: in eval model batch norm doesn’t calculate the statistics of the current batch, that may lead to your issue.</p>