Require_grad in leaf node

username: Ali_Mirzaeyan
cooked: <p>Hi, I don’t understand why this piece of code works not properly. I enabled the “required_grad” for the input dataset, then normalize it, and when I backward through the model (M_O) I expect that “data_batch_.grad.data” has some values other than zero but it always zero. Something is getting wrong in the normalize function which I don’t know what it is. I’m saying that because when I bypass the normalize function the gradients are not zero ( works correctly ).</p>
<pre><code class="lang-auto">
def normalize(t, mean, std, dataset):

    if dataset == 'mnist':
        t[:, 0, :] = (t[:, 0, :] - mean[0]) / std[0]
    if dataset == 'cifar10':
        t[:, 0, :, :] = (t[:, 0, :, :].detach() - mean[0]) / std[0]
        t[:, 1, :, :] = (t[:, 1, :, :].detach() - mean[1]) / std[1]
        t[:, 2, :, :] = (t[:, 2, :, :].detach() - mean[2]) / std[2]

    return t
</code></pre>
<pre><code class="lang-auto">....
    criterion = nn.CrossEntropyLoss()
    criterion = criterion.to(device)
...
     for i, (data_batch, labels_batch) in enumerate(adv_iterator):
                data_batch = data_batch.to(device)
                labels_batch = labels_batch.to(device)

                data_batch_ = normalize(data_batch.clone().detach(), mean, std, chose_dataset)
                **data_batch_ = torch.tensor(data_batch_, requires_grad=True, device=device)**
               # **data_batch_ = data_batch**

                out_adv, _, _ = M_O(data_batch_)
                loss = criterion(out_adv, labels_batch)

                M_O.zero_grad()
                loss.backward()

                noise = torch.sign(data_batch_.grad.data)

                print(noise[0])
...
</code></pre>

username: albanD
cooked: <p>Hi,</p>
<p>First, you should never use <code>.data</code> as it mostly leads to issues.</p>
<p>Second, you call <code>.detach()</code> on the data_batch before passing it to the normalize function. So you explicitly break the graph. This is why you don’t get any gradients back.</p>

username: Ali_Mirzaeyan
cooked: <p>I detached the “data_batch”, however I want the gradient of “data_batch_” ( which is defined based on the normalized “data_batch_”), so the graph should be fine from “data_batch_” to the end (loss function), am I wrong ?</p>

username: albanD
cooked: <p>Ho sorry I got confused because you were mentioning your normalize function. But it is not relevant here since you create your leaf after it?<br>
One thing you can do to be sure is <code>data_batch_.register_hook(print)</code>. This will print the gradient of data_batch_ when it is computed.<br>
If it prints, that means that the gradients are properly computed and they just happen to be 0. most likely because you have non-differentiable ops later in your network.<br>
If it does not print, then it means that your break the graph somewhere below inside your network.</p>

username: Ali_Mirzaeyan
cooked: <p>I don’t know what is going on inside normalize that causes the problem, because like I said when I just skipped the “normalize” function it return correct answer.<br>
I mean if I replace these line of code</p>
<pre><code class="lang-auto">                data_batch_ = normalize(data_batch.clone().detach(), mean, std, chose_dataset)
                data_batch_ = torch.tensor(data_batch_, requires_grad=True, device=device)
               # **data_batch_ = data_batch*
</code></pre>
<p>with just this<br>
<code>data_batch_ = data_batch</code><br>
it shows me some expected results. So I infer that something is off about this normalize function.</p>

username: albanD
cooked: <p>In this case, can you replace the second line with:</p>
<pre><code class="lang-auto">data_batch_ = data_batch_.detach().to(device).requires_grad_()
</code></pre>
<p>To make sure you get a proper leaf?</p>

username: Ali_Mirzaeyan
cooked: <aside class="quote no-group" data-username="albanD" data-post="6" data-topic="82263">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p><code>data_batch_ = data_batch_.detach().to(device).requires_grad_()</code></p>
</blockquote>
</aside>
<p>still the same results</p>

username: albanD
cooked: <p>Can you create a code sample (30/40) lines that I will be able to run that shows this please?</p>

username: Ali_Mirzaeyan
cooked: <p>I changed the normalize code from</p>
<pre><code class="lang-auto">def normalize(t, mean, std, dataset):

    if dataset == 'mnist':
        t[:, 0, :] = (t[:, 0, :] - mean[0]) / std[0]
    if dataset == 'cifar10':
        t[:, 0, :, :] = (t[:, 0, :, :].detach() - mean[0]) / std[0]
        t[:, 1, :, :] = (t[:, 1, :, :].detach() - mean[1]) / std[1]
        t[:, 2, :, :] = (t[:, 2, :, :].detach() - mean[2]) / std[2]

    return t
</code></pre>
<p>to this</p>
<pre><code class="lang-auto">def normalize(t, mean, std, dataset):
    return t
</code></pre>
<p>and It is working, so I assuming that I should rewrite the code inside the normalize function differently.</p>