Learning the image to maximize goldfish category

blackbirdbarber: <p>I created this code but something is wrong, the input image should learn the best goldfish representation because I use the goldfish ImageNet category which is second category. Starting from the random image I should update the input image for a number of iterations in code.<br>
However, the end result is I get the loss down to zero, but the image still looks like a random noise. Can you please find where I am wrong:</p>
<pre><code class="lang-auto">import torch
import torchvision.models as models
import torch.optim as optim
from PIL import Image
import numpy as np
import torch.nn.functional as F  # Import functional interface for torch.nn
from torchvision.models.vgg import VGG16_Weights

# Load a pre-trained VGG16 model
vgg16 = models.vgg16(weights=VGG16_Weights.IMAGENET1K_V1)
vgg16.eval()  # Set the model to evaluation mode

# Index of the target class (goldfish in this case)
target_class = 1  # Index 1 corresponds to 'goldfish' in ImageNet classes

# Define the optimizer (to update the input image)
input_image = torch.randn(1, 3, 224, 224).requires_grad_(True)  # Initialize input image with random noise
optimizer = optim.SGD([input_image], lr=0.05)  # Use the input image as parameter to optimize

# Number of optimization steps
num_iterations = 1000

# Optimization loop
for i in range(num_iterations):
    print(i)
    optimizer.zero_grad()  # Zero gradients
    
    # Forward pass through VGG16 with the current input image
    output = vgg16(input_image)
    
    # Apply Softmax function to the output tensor
    probabilities = F.softmax(output, dim=1)
    
    # Compute the loss - maximize the probability of the target class
    loss = -probabilities[0, target_class].log()  # Negative log probability of the target class
    print(loss)
    
    # Backpropagation: Compute gradients of the input image wrt the loss
    loss.backward()
    
    # Update the input image using the gradients (gradient ascent)
    optimizer.step()
    
    # Clamp the image pixel values to stay within valid range (0-1)
    #input_image.data.clamp_(0, 1)

    if i % 10 == 0:
        # Convert the optimized input image tensor to a PIL image
        generated_image = input_image.squeeze(0).detach().cpu().numpy()
        generated_image = np.moveaxis(generated_image, 0, -1)  # Change from [C, H, W] to [H, W, C]
        generated_image = (generated_image * 255).astype(np.uint8)  # Rescale to [0, 255] for PIL

        # Save the generated image
        image_output = Image.fromarray(generated_image)
        image_output.save("generated_goldfish_image" + str(i) + ".jpg")

</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/6/16ad4bb733965fda0e0514421f683cb44de632c1.jpeg" alt="image" data-base62-sha1="3eBMUc3wQ2kch5QeX26Cu7YfAyd" width="320" height="312"></p>

KFrank: <p>Hi Blackbird!</p>
<aside class="quote no-group quote-modified" data-username="blackbirdbarber" data-post="1" data-topic="195064" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/b/6de8d8/48.png" class="avatar"> blackbirdbarber:</div>
<blockquote>
<p>the input image should learn the best goldfish representation<br>
…<br>
but the image still looks like a random noise.</p>
</blockquote>
</aside>
<p>The short story:  To create a synthetic image that looks like a goldfish,<br>
you should train a <a href="https://pytorch.org/tutorials/beginner/dcgan_faces_tutorial.html" rel="noopener nofollow ugc"><em>generative adversarial network</em>.</a></p>
<p>I haven’t looked at your code, so maybe there’s a bug in it somewhere.</p>
<p>However, there’s an important conceptual issue here.</p>
<p>Your pretrained <code>vgg16</code> has been trained so that if you present it with a<br>
picture of a goldfish, it should classify it as goldfish rather than, say, a<br>
magpie or a scorpion.  You would also expect that if you present it with an<br>
image of a coronavirus, it will not predict any class with a high probability.<br>
(Some classes will have higher probabilities and some lower, but none<br>
should be predicted to be very likely or certain.)  Likewise, if you present<br>
the model with a random-noise image, you should get no clear prediction.</p>
<p>However, the <code>vgg16</code> as <em>not</em> been trained to not be “fooled” by a non-random<br>
image (that happens to look like random noise to the human eye) that you<br>
have carefully constructed to “look like” a goldfish (in some weird way) to your<br>
specific pretrained <code>vgg16</code>.</p>
<p>Figuratively speaking, the image you have constructed has some pixels that<br>
look like a goldfish texture, some that look like goldfish coloring, some that<br>
look like the ghost of a goldfish eye, and so on, but even though all of these<br>
goldfish cues don’t fit together into anything resembling a goldfish, your<br>
model sees enough of these goldfish cues (and no magpie cues, etc.) and<br>
predicts “goldfish” with high probability.</p>
<p>And furthermore, you’ve constructed your image to contain those particular<br>
goldfish cues that <em>this particular</em> <code>vgg16</code> cares about.</p>
<p>Consider the following experiment:  Train two <code>vgg16</code>s from scratch separately,<br>
starting with two different random initializations.  Train on the same dataset<br>
(but randomly shuffle your batches differently, if you so choose).  Use the<br>
first <code>vgg16</code> to construct your trick image.  It will look like random noise, but<br>
the <em>first</em> <code>vgg16</code> will classify it as a goldfish.  However, the <em>second</em> <code>vgg16</code><br>
will not predict any specific class with high probability – that is, it will look<br>
like random noise to the second <code>vgg16</code>.</p>
<p>However, when you train a generative adversarial network, you are both<br>
training a <em>generator</em> network to produce images that look real (both to<br>
the human eye and another network) while <em>simultaneously</em> training a<br>
<em>discriminator</em> network to correctly classify real images while not being<br>
“fooled” by your synthetic images.</p>
<p>Initially, your generator will learn to constructed the specific goldfish-like<br>
cues (that don’t really look like a goldfish) that the <em>current version</em> of the<br>
discriminator recognizes as “goldfish.”  But as you train the discriminator,<br>
future versions of it have learned that those particular cues were synthetic,<br>
so it doesn’t recognize those cues as “goldfish” any more.  This process<br>
goes back and forth until your generator learns to construct images that<br>
really do look like goldfish and whose cues can no longer be recognized<br>
as being synthetically generator.</p>
<p>Best.</p>
<p>K. Frank</p>

blackbirdbarber: <p>Hi KFrank,</p>
<p>Thank you for the feedback very much.<br>
It was not my intention to create the adversarial network to do feature visualization at this time.</p>
<p>The code I shared is capable of running in a colab notebook, and it should take few minutes to execute. The process of updating the input image is present because I check files being created with MD5 checksum.</p>
<p>I even tried to start from a goldfish image and to maximize the end category by updating the input image. I expected the input image will have even more goldfish artifacts at the end, for instance several fish eyes, fish scales, etc.</p>
<p>However, even thought in each step the image is different from MD5 perspective it looks the same as the original image.</p>
<p>The inspiration I took from the great article</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://distill.pub/2017/feature-visualization/">
  <header class="source">
      <img class="site-icon">

      <a href="https://distill.pub/2017/feature-visualization/" target="_blank" rel="noopener nofollow ugc">Distill</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:560/295;"><img src="https://distill.pub/2017/feature-visualization/thumbnail.jpg" class="thumbnail" width="690" height="363"></div>

<h3><a href="https://distill.pub/2017/feature-visualization/" target="_blank" rel="noopener nofollow ugc">Feature Visualization</a></h3>

  <p>How neural networks build up their understanding of images</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>And particularly I am using this image and the latest example where I set the category of goldfish after softmax.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/3/636b9033947cee0e83f8dc48ac7af1fbbd23f2c8.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/636b9033947cee0e83f8dc48ac7af1fbbd23f2c8" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/636b9033947cee0e83f8dc48ac7af1fbbd23f2c8_2_345x168.jpeg" alt="image" data-base62-sha1="ebvIOEnllF4erOATZ6NB4X6oTSo" width="345" height="168" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/636b9033947cee0e83f8dc48ac7af1fbbd23f2c8_2_345x168.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/636b9033947cee0e83f8dc48ac7af1fbbd23f2c8_2_517x252.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/3/636b9033947cee0e83f8dc48ac7af1fbbd23f2c8_2_690x336.jpeg 2x" data-dominant-color="CECBC2"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1090×531 67 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It should work, but I don’t know why in my case is not working, maybe I have some bug, and maybe proper initialization is missing.</p>