Conditional GAN with multiple adversarial losses

Frankie91: <p>I would like to test the architecture from the following paper with a different dataset:</p>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://www.mdpi.com/2072-4292/13/19/3834">
  <header class="source">

      <a href="https://www.mdpi.com/2072-4292/13/19/3834" target="_blank" rel="noopener nofollow ugc">MDPI</a>
  </header>

  <article class="onebox-body">
    <img width="" height="" src="https://pub.mdpi-res.com/remotesensing/remotesensing-13-03834/article_deploy/html/images/remotesensing-13-03834-g001-550.jpg?1632744070" class="thumbnail">

<h3><a href="https://www.mdpi.com/2072-4292/13/19/3834" target="_blank" rel="noopener nofollow ugc">Improved 1-km-Resolution Hourly Estimates of Aerosol Optical Depth Using...</a></h3>

  <p>Aerosol Optical Depth (AOD) is a crucial parameter for various environmental and climate studies. Merging multi-sensor AOD products is an effective way to produce AOD products with more spatiotemporal integrity and accuracy. This study proposed a...</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>The authors state that their objective function is the following:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6fb0476afad15e2feaf1893a543d98ccacb6034.png" data-download-href="https://discuss.pytorch.org/uploads/default/b6fb0476afad15e2feaf1893a543d98ccacb6034" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6fb0476afad15e2feaf1893a543d98ccacb6034_2_690x50.png" alt="image" data-base62-sha1="q6IHwtNahPsbFeOj8a7ETrjekfy" width="690" height="50" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/6/b6fb0476afad15e2feaf1893a543d98ccacb6034_2_690x50.png, https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6fb0476afad15e2feaf1893a543d98ccacb6034.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/b/6/b6fb0476afad15e2feaf1893a543d98ccacb6034.png 2x" data-dominant-color="F1F1F2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1027×75 12.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Where:</p>
<p>-<strong>x</strong> is the real grayscale image.</p>
<p>-<strong>s</strong> is its downsampled version, which should be used both as the initial imput of the generator performing the super-resolution and as a first conditional variable in the learning process.</p>
<p>-<strong>e</strong> is another two-dimensional array containing values for a second additional conditional variable.</p>
<p>The authors, however, state that this should be implemented by using two separate conditional adversarial losses, one for each of the conditional variables.</p>
<p>To clarify, the first adversarial loss should be:</p>
<p>AdvLoss1(ParametersG, ParametersD) = - Log(Discriminator(<strong>x</strong>,<strong>s</strong>) - Log(1-Discriminator(Generator(<strong>s</strong>),<strong>s</strong>)</p>
<p>While the second would be:</p>
<p>AdvLoss2(ParametersG, ParametersD) = - Log(Discriminator(<strong>x</strong>,<strong>e</strong>) - Log(1-Discriminator(Generator(<strong>s</strong>),<strong>e</strong>)</p>
<p>Which should be then summed up for the backward pass.<br>
In my pytorch implementation, however, I have only been able to set up a unique adversarial loss, which could be defined as:</p>
<p>CurrentAdvLoss(ParametersG, ParametersD) = - Log(Discriminator(x,(<strong>s</strong>,<strong>e</strong>)) - Log(1-Discriminator(Generator(<strong>s</strong>),(<strong>s</strong>,<strong>e</strong>))</p>
<p>which I calculate in the following training loop (simplified version) as <em>errD</em> and <em>errG</em> after conditioning the network on both <strong>s</strong> and <strong>e</strong> at the same time:</p>
<pre><code class="lang-auto">for epoch in range(num_epochs):
    for i, data in enumerate(trainloader, 0):
        ############################
        # (1) Update D network
        ###########################
        ## Train with all-real batch
            Discriminator.zero_grad()
            s = data[0].to(device)
            x = data[1].to(device)
            e = data[2].to(device)            
            x_fake = Generator(s,e)
            # Forward pass real batch through D
            output_real = Discriminator(x,x_fake,s,e).view(-1)
            label_real = torch.ones(len(x), dtype=torch.float, device=device)            
            # Calculate loss on all-real batch
            errD_real = nn.BCELoss(output_real, label_real)
            errD_real.backward() 
    
            # Train with all-fake batch 
            x_fake = Generator(s,e)
            # Generate fake image batch with G
            output_fake = Discriminator(x.detach(), x_fake.detach(),
                               s.detach(), e.detach()).view(-1)
            label_fake = torch.zeros(len(x_fake), dtype=torch.float, device=device)
            # Calculate D's loss on the all-fake batch
            errD_fake = nn.BCELoss(output_fake, label_fake)
            # Calculate the gradients for this batch, accumulated (summed) with previous gradients
            errD_fake.backward()
            # Compute error of D as sum over the fake and the real batches
            errD = errD_real + errD_fake
            # Update D
            optimizerD.step()
    
            ############################
            # (2) Update Generator
            ###########################
            Generator.zero_grad()
            # Since we just updated D, perform another forward pass of all-fake batch through D
            output_fake = Discriminator(x, x_fake,s, e).view(-1)
            label_gen = torch.ones(len(x_fake), dtype=torch.float, device=device)
            # Calculate G's loss based on this output
            errG = nn.BCELoss(output_fake, label_gen)
            # Calculate gradients for G
            errG.backward()
            # Update G
            optimizerG.step()
</code></pre>
<p>My question is, is there a way to modify the following loop to obtain outputs that have been separately conditioned only first on <strong>s</strong> and then on <strong>e</strong> and thus calculate the two separate adversarial losses originally proposed by the authors instead?</p>