Why don't we put models in .train() or .eval() modes in DCGAN example

devansh20la: <p>I’m trying to figure out, why don’t we put generator model in eval() mode at the end when using fixed_noise as input or when we are just training the discriminator. I believe the batch norm layers should behave differently evaluation mode.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/examples/blob/master/dcgan/main.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/examples/blob/master/dcgan/main.py" target="_blank" rel="nofollow noopener">pytorch/examples/blob/master/dcgan/main.py</a></h4>
<pre><code class="lang-py">from __future__ import print_function
import argparse
import os
import random
import torch
import torch.nn as nn
import torch.nn.parallel
import torch.backends.cudnn as cudnn
import torch.optim as optim
import torch.utils.data
import torchvision.datasets as dset
import torchvision.transforms as transforms
import torchvision.utils as vutils
from torch.autograd import Variable


parser = argparse.ArgumentParser()
parser.add_argument('--dataset', required=True, help='cifar10 | lsun | imagenet | folder | lfw | fake')
parser.add_argument('--dataroot', required=True, help='path to dataset')
parser.add_argument('--workers', type=int, help='number of data loading workers', default=2)
</code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/examples/blob/master/dcgan/main.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>I tried training with eval mode but the model collapses to one particular image. <img src="//discuss.pytorch.org/images/emoji/apple/confused.png?v=5" title=":confused:" class="emoji" alt=":confused:"></p>
<p>Thank you</p>

zhangboknight: <p>I am confused about this too. Did you solve the problem?</p>

smth: <p>because GAN training is highly unstable, the <code>.eval()</code> mode is not as good as <code>.train()</code> mode particularly for DCGAN.</p>
<p>For a model to be stable and good in <code>eval()</code> mode, you first have to stop training and do generations for a few mini-batches, so that the <code>running_mean</code> and <code>running_std</code> of batchnorm stabilize.</p>

haoyangz: <p>Hi <a class="mention" href="/u/smth">@smth</a>  just trying to make it crystal, is this what you meant?</p>
<pre><code class="lang-auto"># Training phase
G.train()
D.train()
for epoch in range(n_epoch):
    ....  # train D and G

# Now run generation for a few min-batch while keeping the mode as "train"
for _ in range(additional_iters):
    z = ...  # sample a z
    _ = G(z)

# Now switch to eval mode to do the actual generation
G.eval()
D.eval()
z = ...
samples2output = G(z)
</code></pre>

smth: <p><a class="mention" href="/u/haoyangz">@haoyangz</a> yes. that’s what I meant.</p>

LilFish: <p>Could you explain why we still need to manually get running_mean and running_std in for eval() mode? Does pytorch already keep a running_mean and running_std during train() and use them for eval()?</p>
<p>My cycleGAN model does not use dropout but uses instancenorm. It performs badly when switching to eval() mode.</p>

smth: <p>read my comment above (two comments above), where I explain why.</p>

LilFish: <p>I understand that we need a stable running_mean and running_std for eval(). But I thought pytorch already keeps a running_mean and running_std during train() and use them for eval(). Is that not the case? I’m a bit confused on why we need to get a running_mean and running_std for eval() separately. We cannot use the running_mean and running_std from train()?</p>
<p>Also, I am using InstanceNorm2D which is suppose to perform the same in train() and eval()?<br>
<aside class="onebox githubcommit">
  <header class="source">
      <a href="https://github.com/gchanan/pytorch/commit/5a17375eaa3e23b9224b769c7f93e6799fef80c9" target="_blank" rel="nofollow noopener">github.com/gchanan/pytorch</a>
  </header>
  <article class="onebox-body">
      <a href="https://github.com/DmitryUlyanov" target="_blank" rel="nofollow noopener">
    <img alt="DmitryUlyanov" src="https://avatars2.githubusercontent.com/u/6727524?v=4" class="thumbnail onebox-avatar" width="90" height="90">
  </a>

<h4>
  <a href="https://github.com/gchanan/pytorch/commit/5a17375eaa3e23b9224b769c7f93e6799fef80c9" target="_blank" rel="nofollow noopener">fixes eval mode in InstanceNorm (#1604)</a>
</h4>

  <pre class="message" style="white-space: normal;">fixes https://github.com/pytorch/pytorch/issues/1541</pre>

<div class="date">
  by <a href="https://github.com/DmitryUlyanov" target="_blank" rel="nofollow noopener">DmitryUlyanov</a>
  on <a href="https://github.com/gchanan/pytorch/commit/5a17375eaa3e23b9224b769c7f93e6799fef80c9" target="_blank" rel="nofollow noopener">05:27PM - 21 May 17</a>
</div>

<div class="github-commit-stats">
  changed <strong>1 files</strong>
  with <strong>1 additions</strong>
  and <strong>1 deletions</strong>.
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>Thanks!</p>

smth: <blockquote>
<p>because GAN training is highly unstable, the <code>.eval()</code> mode is not as good as <code>.train()</code> mode particularly for DCGAN.</p>
</blockquote>

John1231983: <p>Very interesting! Thanks for sharing the tip<br>
I have tried it in my code and it does not much different when using eval during training. Just one question, do we need train G in the additional_iters <a class="mention" href="/u/smth">@smth</a> ? Thanks</p>
<pre><code class="lang-auto"># Now run generation for a few min-batch while keeping the mode as "train"
 for _ in range(additional_iters): 
      z = ... # sample a z
       _ = G(z)
      # Do we need train G here without train D?
</code></pre>

Eoin_Kenny: <p>I’m also curious about this, I mean you must have to train the GAN during the addtional iterations right? Otherwise how could the batchnorm parameters change?</p>
<p>Is another solution to just train the GAN for a few epochs at the end with a batch size of 1?</p>