3 ways of calculating gradient give 3 differents results

AlexisAlexis: <p>Hello,<br>
To make a kind of fast gradient sign attack, I try to calculate the gradient of an image against a softmax loss. The problem is that the image is normalized before entering the model, and I want to get the gradient relative to the unnormalized (untouched) image.</p>
<p>I think the theoretical gradient of the normalization transform:<br>
<code>torchvision.transforms.Normalize(mean=mean, std=std)</code><br>
can be obtained with:<br>
<code>torchvision.transforms.Normalize(mean=0, std=std)</code></p>
<p>I tried to calculate the gradient in three different ways but I get 3 different results. What am I doing wrong?</p>
<p>The model I use is a model transferred from AlexNet :</p>
<pre><code class="lang-auto">  n_classes = 10
  model = torchvision.models.alexnet(pretrained=False)
  model.classifier[6] = torch.nn.Linear(model.classifier[6].in_features, n_classes)
  model.load_state_dict(torch.load(path_model, map_location=torch.device('cpu')))
</code></pre>
<pre><code class="lang-auto">    image = PIL.Image.open("U:\\PROJET_3A\\imagenette2-160\\train\\n01440764\\n01440764_17174.JPEG")
    transform = torchvision.transforms.Compose([torchvision.transforms.Resize(256),
                                                torchvision.transforms.CenterCrop(224),
                                                torchvision.transforms.ToTensor(),])
    image = transform(image)
    image = image[None, :]

    mean = [0.485, 0.456, 0.406]
    std = [0.229, 0.224, 0.225]

    normalize = torchvision.transforms.Normalize(mean=mean, std=std)
    grad_of_normalization = torchvision.transforms.Normalize(mean=0, std=std)
    target_class = 1

    test1 = image.clone().detach()
    test1.requires_grad = True
    vector_scores = self.model(normalize(test1))  
    loss_target = -torch.nn.functional.log_softmax(vector_scores, dim=1)[0, target_class]
    loss_target.backward()
    grad_test1 = test1.grad.clone().detach()

    test2 = torch.autograd.Variable(normalize(image.clone().detach()), requires_grad=True)
    vector_scores = self.model(test2)  
    loss_target = -torch.nn.functional.log_softmax(vector_scores, dim=1)[0, target_class]
    loss_target.backward()
    grad_test2 = test2.grad.clone().detach()
    grad_test2 = grad_of_normalization(grad_test2)


    test3 = normalize(image).clone().detach()
    test3.requires_grad = True
    vector_scores = self.model(test3)  
    loss_target = -torch.nn.functional.log_softmax(vector_scores, dim=1)[0, target_class]
    loss_target.backward()
    grad_test3 = test3.grad.clone().detach()
    grad_test3 = grad_of_normalization(grad_test3)
</code></pre>

AlphaBetaGamma96: <p>Hi <a class="mention" href="/u/alexisalexis">@AlexisAlexis</a>,</p>
<p>Can you call <code>self.model.zero_grad()</code> in-between measurements? Gradients are accumulated in <code>.backward()</code>, so the result for your 2nd method is actually showing method 1 + method 2.</p>
<p>Also, can you share the results? How different are they?</p>

AlexisAlexis: <p>Hi,<br>
I added model.zero_grad() between the measurements and then displayed the gradients without modifying them with matplotlib. The differences are not huge but I see more intense spots in some measurements.</p>
<p>Example :<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/a/da00c534fd2350ce1db0d21bc8d0a5fa4a8efeb5.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/da00c534fd2350ce1db0d21bc8d0a5fa4a8efeb5" title="grads"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da00c534fd2350ce1db0d21bc8d0a5fa4a8efeb5_2_690x172.jpeg" alt="grads" data-base62-sha1="v6xJOJaXHoLVHhDwvDMjg4LPSDj" width="690" height="172" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da00c534fd2350ce1db0d21bc8d0a5fa4a8efeb5_2_690x172.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/a/da00c534fd2350ce1db0d21bc8d0a5fa4a8efeb5_2_1035x258.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/d/a/da00c534fd2350ce1db0d21bc8d0a5fa4a8efeb5.jpeg 2x" data-dominant-color="5D5D5A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">grads</span><span class="informations">1267×317 43.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

AlphaBetaGamma96: <p>Can you compare the error via its MSE or MAE value instead of a figure?</p>

AlexisAlexis: <p>I added the calculation of the MSE, here are my results:</p>
<pre><code class="lang-auto">mse grad 1 grad 2 0.000966
mse grad 1 grad 3 0.000919
mse grad 2 grad 3 0.001061
</code></pre>
<p>Here is a part of my code after modification, don’t hesitate to tell me if something is wrong:</p>
<pre><code class="lang-auto">test1 = image.clone().detach()
test1.requires_grad = True
vector_scores = self.model(normalize(test1))  
loss_target = -torch.nn.functional.log_softmax(vector_scores, dim=1)[0, target_class]
loss_target.backward()
grad_test1 = test1.grad.clone().detach()

self.model.zero_grad()

test2 = torch.autograd.Variable(normalize(image.clone().detach()), requires_grad=True)
vector_scores = self.model(test2)  
loss_target = -torch.nn.functional.log_softmax(vector_scores, dim=1)[0, target_class]
loss_target.backward()
grad_test2 = test2.grad.clone().detach()
grad_test2 = grad_of_normalization(grad_test2)

self.model.zero_grad()

test3 = normalize(image.clone().detach())
test3.requires_grad = True
vector_scores = self.model(test3)  
loss_target = -torch.nn.functional.log_softmax(vector_scores, dim=1)[0, target_class]
loss_target.backward()
grad_test3 = test3.grad.clone().detach()
grad_test3 = grad_of_normalization(grad_test3)

print("mse grad 1 grad 2 %f" % torch.nn.functional.mse_loss(grad_test1, grad_test2))
print("mse grad 1 grad 3 %f" % torch.nn.functional.mse_loss(grad_test1, grad_test3))
print("mse grad 2 grad 3 %f" % torch.nn.functional.mse_loss(grad_test2, grad_test3))
</code></pre>

AlexisAlexis: <p>Solution : I forgot to set the model in eval mode with .eval()</p>