Understanding the ResnetGenerator in the CycleGan Model

username: Imahn
cooked: <p>In their famous research paper on CycleGans (<a href="https://arxiv.org/pdf/1703.10593v7.pdf" rel="noopener nofollow ugc">https://arxiv.org/pdf/1703.10593v7.pdf</a>), the authors implement - well, a CycleGan.</p>
<p>There are two discriminators and two generators for the CycleGan. Now, they also provide their neural networks code. I am particularly interested in the generator(s), which they implement here:</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/models/networks.py#L119-L159" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/models/networks.py#L119-L159" target="_blank" rel="noopener nofollow ugc">junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/models/networks.py#L119-L159</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="119" style="counter-reset: li-counter 118 ;">
<li>def define_G(input_nc, output_nc, ngf, netG, norm='batch', use_dropout=False, init_type='normal', init_gain=0.02, gpu_ids=[]):</li>
<li>    """Create a generator</li>
<li>
<li>    Parameters:</li>
<li>        input_nc (int) -- the number of channels in input images</li>
<li>        output_nc (int) -- the number of channels in output images</li>
<li>        ngf (int) -- the number of filters in the last conv layer</li>
<li>        netG (str) -- the architecture's name: resnet_9blocks | resnet_6blocks | unet_256 | unet_128</li>
<li>        norm (str) -- the name of normalization layers used in the network: batch | instance | none</li>
<li>        use_dropout (bool) -- if use dropout layers.</li>
<li>        init_type (str)    -- the name of our initialization method.</li>
<li>        init_gain (float)  -- scaling factor for normal, xavier and orthogonal.</li>
<li>        gpu_ids (int list) -- which GPUs the network runs on: e.g., 0,1,2</li>
<li>
<li>    Returns a generator</li>
<li>
<li>    Our current implementation provides two types of generators:</li>
<li>        U-Net: [unet_128] (for 128x128 input images) and [unet_256] (for 256x256 input images)</li>
<li>        The original U-Net paper: https://arxiv.org/abs/1505.04597</li>
<li>
</ol></code></pre>

  This file has been truncated. <a href="https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix/blob/master/models/networks.py#L119-L159" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<p>So they use a <code>ResnetGenerator</code>, but I’m afraid I do not really understand it yet (cf. lines 119-159 and 315-373).</p>
<p>For the generator, why do we have both downsampling (<code>Conv2d</code>) and upsampling (<code>ConvTranpose2d</code>) layers? I generally know it like this for the generator that the generator <strong>only</strong> uses <code>ConvTranpose2d</code> layers, where the input is noise sampled from a uniform or Gaussian distribution…</p>
<p>That’s why I am confused…</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Imahn" data-post="1" data-topic="117054">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/i/8baadc/48.png" class="avatar"> Imahn:</div>
<blockquote>
<p>For the generator, why do we have both downsampling (<code>Conv2d</code>) and upsampling (<code>ConvTranpose2d</code>) layers? I generally know it like this for the generator that the generator <strong>only</strong> uses <code>ConvTranpose2d</code> layers, where the input is noise sampled from a uniform or Gaussian distribution…</p>
</blockquote>
</aside>
<p>Based on Section7.1 from the paper the authors are reusing the <code>image transformation network</code> from <a href="https://arxiv.org/pdf/1603.08155.pdf">Perceptual Losses for Real-Time Style Transfer and Super-Resolution</a>, which uses this bottleneck architecture. I can’t find more details about this choice so I would assume that this model architecture worked fine for their implementation.</p>