How to backprop the min of loss A and max of loss B?

username: machina
cooked: <p>In the process of training a GAN, I am trying to change the params of gen() such that a cross-entropy loss: <code>f_loss = CE_loss(f_pred, labels)</code> is maximized and a different type of loss: <code>lossG = torch.mean(torch.log(1. - output))</code> is minimized.</p>
<p>This is my code:</p>
<pre><code class="lang-auto">adv_ex = adv_ex.reshape(32, 28*28)
output = disc(adv_ex) #discriminator decides if advex is real or fake
lossG = torch.mean(torch.log(1. - output)) #get loss for gen's desired desc pred

adv_ex = adv_ex.reshape(-1,1,28,28)
f_pred = target(adv_ex) #.size() = [32, 10]
f_loss = CE_loss(f_pred, labels) #add loss for gens desired f pred
print(f_loss)
loss_G_Final = f_loss+lossG # can change the weight of this loss term later
     
opt_gen.zero_grad()
loss_G_Final = loss_G_Final.to(device)
loss_G_Final.backward()
opt_gen.step()
</code></pre>
<p>I’m pretty sure that this will minimize lossG, but also minimize f_loss.  How can I change this so that the gen() parameters will maximize the crossentropy f_loss as well as minimize lossG?</p>
<p>here is the entire code block if needed:</p>
<pre><code class="lang-auto">
# set up the optimizers and loss for the models
opt_disc = optim.Adam(disc.parameters(), lr=lr)
opt_gen = optim.Adam(gen.parameters(), lr=lr) 
CE_loss = nn.CrossEntropyLoss()

for epoch in range(num_epochs):
    for batch_idx, (real, labels) in enumerate(loader):
        #get a fixed input batch to display gen output
        if batch_idx == 0:
            if epoch == 0:
                fixed_input = real.view(-1,784).to(device)
        
        adv_ex = real.clone().reshape(-1,784).to(device) # [32, 784] advex copy of first batch flattened
        real = real.view(-1, 784).to(device) # [32, 784] # real batch flattened
        labels = labels.to(device) # size() [32] 32 labels in batch
        
        
        #purturb each image in adv_ex
        tmp_adv_ex = []
        for idx, item in enumerate(adv_ex):
            purturbation = gen(adv_ex[idx])
            tmp_adv_ex.append(adv_ex[idx] + purturbation)
        adv_ex = torch.cat(tmp_adv_ex, dim=0)
        
        
         
        # Train Generator: min log(1 - D(G(z))) &lt;-&gt; max log(D(G(z))
        # also max the CE loss between target-model pred and true labels

        
        adv_ex = adv_ex.reshape(32, 28*28)
        output = disc(adv_ex) #discriminator decides if advex is real or fake
        lossG = torch.mean(torch.log(1. - output)) #get loss for gen's desired desc pred

        adv_ex = adv_ex.reshape(-1,1,28,28)
        f_pred = target(adv_ex) #.size() = [32, 10]
        f_loss = CE_loss(f_pred, labels) #add loss for gens desired f pred
        print(f_loss)
        loss_G_Final = f_loss+lossG # can change the weight of this loss term later
        
        opt_gen.zero_grad()
        loss_G_Final = loss_G_Final.to(device)
        loss_G_Final.backward()
        opt_gen.step()
        
        # Train Discriminator: max log(D(x)) + log(1 - D(G(z)))
        
        adv_ex = adv_ex.reshape(32, 784)
        disc_real = disc(real).view(-1)
        disc_fake = disc(adv_ex).view(-1)
        lossD = -torch.mean(torch.log(disc(real)) + torch.log(1. - disc(adv_ex)))

        
        opt_disc.zero_grad()
        lossD.backward()
        opt_disc.step()
</code></pre>

username: KFrank
cooked: <p>Hi Ryan!</p>
<aside class="quote no-group quote-modified" data-username="machina" data-post="1" data-topic="132982" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ad7895/48.png" class="avatar"> machina:</div>
<blockquote>
<p>I am trying to change the params of gen() such that a cross-entropy loss: <code>f_loss = CE_loss(f_pred, labels)</code> is maximized and a different type of loss: <code>lossG = torch.mean(torch.log(1. - output))</code> is minimized.</p>
<pre><code class="lang-auto">f_pred = target(adv_ex) #.size() = [32, 10]
f_loss = CE_loss(f_pred, labels) #add loss for gens desired f pred
print(f_loss)
loss_G_Final = f_loss+lossG # can change the weight of this loss term later
     
opt_gen.zero_grad()
loss_G_Final = loss_G_Final.to(device)
loss_G_Final.backward()
opt_gen.step()
</code></pre>
</blockquote>
</aside>
<p>I haven’t looked at you code in detail, and I’m not sure I understand<br>
the signs of your losses.</p>
<p>Let me say what I <em>think</em> you’re trying to do.</p>
<p>First, when a model is doing “better,” its associated loss is lower.<br>
Pytorch’s optimizers, such a <code>Adam</code>, seek to <em>minimize</em> the loss.</p>
<p>I assume that <code>f_loss</code> is measuring how well your generator, <code>gen</code>,<br>
is generating a “fake” sample with certain desired characteristics.<br>
If a lower value of <code>f_loss</code> means that <code>gen</code> is doing better, then<br>
calling <code>f_loss.backward()</code> and <code>opt_gen.step()</code> will move <code>gen</code>’s<br>
parameters in the direction that improves <code>gen</code>’s performance (as<br>
measured by <code>f_loss</code>).</p>
<p>Now I assume that <code>lossG</code> measures how good your discriminator,<br>
<code>disc</code>, is at telling the “fake” samples generated by <code>gen</code> from real<br>
samples.  Smaller values of <code>lossG</code> mean that your discriminator<br>
is working better.</p>
<p>The complexity comes because you want to train <code>disc</code> to do well,<br>
that is to lower <code>lossG</code>.  But you want to train <code>gen</code> so that <code>disc</code><br>
has a hard time distinguishing fake from real, that is to <em>increase</em><br>
<code>lossG</code></p>
<p>Assuming that <code>disc</code>’s parameters are (temporarily) frozen, you want<br>
to train <code>gen</code> to lower <code>f_loss</code> while increasing <code>lossG</code>.  Doing this is as<br>
simple as training <code>gen</code> with (note the minus sign):</p>
<pre data-code-wrap="python"><code class="lang-python">loss_G_Final = f_loss - lossG
</code></pre>
<p>(You may well choose to weight <code>f_loss</code> and <code>lossG</code> differently to<br>
adjust <code>gen</code>’s trade-off in producing the characteristics measured<br>
by <code>f_loss</code> vs. tricking <code>disc</code>, as measured by <code>lossG</code>.)</p>
<p>One final note:  When you train <code>disc</code> to successfully distinguish<br>
fake from real, you will <em>not</em> want to flip the sign of <code>lossG</code>.  You<br>
would call:</p>
<pre data-code-wrap="python"><code class="lang-python">lossG.backward()
opt_disc.step()
</code></pre>
<p><code>f_loss</code> doesn’t depend on <code>disc</code>’s parameters (unless your code is<br>
hiding something somewhere), so you neither need nor want to call<br>
<code>f_loss.backward()</code> (nor <code>loss_G_Final.backward()</code>) before calling<br>
<code>opt_disc.step()</code>.</p>
<p>Best.</p>
<p>K. Frank</p>

username: machina
cooked: <p>Thanks for the response!</p>
<p>Ideally, F_Loss would be a measure of how well a classifier: F performs on the adversarial examples: x +G(x).<br>
I want the generator: G to generate noise such that F_Loss is HIGH. This would indicate that the noise: G(x) when added to the image: x will cause F to make a misclassification.</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/b/3b924750cd1c56001242a073fa8a8bc1a5b47f91.png" alt="image" width="604" height="291"></p>
<p>I also want the generator: G to generate noise: G(x) such that the discriminator: D classifies G(x) + x as a real member of x. ie D(G(x) + x) == 1.</p>
<p>D on the other hand should classify G(x) + x as a fake image ie D(G(x)+x) == 0</p>
<p>The goal of this network is to train G to produce adversarial examples that look like real data, and fool F.</p>

username: KFrank
cooked: <p>Hi Ryan!</p>
<aside class="quote no-group" data-username="machina" data-post="3" data-topic="132982" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/m/ad7895/48.png" class="avatar"> machina:</div>
<blockquote>
<p>The goal of this network is to train G to produce adversarial examples that look like real data, and fool F.</p>
</blockquote>
</aside>
<p>As I understand it, your have three models here:  G is <code>gen</code> in the<br>
code you posted above, D is <code>disc</code>, and F, I suppose, is <code>target</code>.</p>
<p>For this part of the training I’ll assume that the parameters of <code>gen</code><br>
and <code>target</code> are (temporarily) frozen, and that you are training just<br>
the parameters of <code>gen</code>.</p>
<p>You want the model <code>target</code> to misclassify, that is (going back to the<br>
code you posted), <code>f_loss</code> is large.  You also want <code>disc</code> to mistake<br>
fake for real, that is, <code>lossG</code> is also large.</p>
<p>Pytorch optimizers seek to minimize the losses they’re given, so you<br>
should simply flip the signs of <code>f_loss</code> and <code>lossG</code> and backpropagate<br>
normally through <code>gen</code>.  (Based on your clarification that you want <code>gen</code><br>
to fool <code>target</code> (as well as <code>disc</code>), you should – unlike what I said in my<br>
first reply – also flip the sign of <code>f_loss</code>.)  Thus:</p>
<pre data-code-wrap="python"><code class="lang-python">loss_G_Final = - (f_loss + lossG)
opt_gen.zero_grad()
loss_G_Final.backward()
opt_gen.step()
</code></pre>
<p>(And again, either in separate training steps or in a combined<br>
procedure, you presumably also want to be training <code>disc</code> to do<br>
as well it can on the fake-vs-real task you are giving it, even as<br>
you train <code>gen</code> to keep getting better at tricking <code>disc</code>.)</p>
<p>Best.</p>
<p>K. Frank</p>