Greedy optimisation with random noise in gradients

Prasad_Raghavendra: <p>I am trying to write code for simple objective: I have usual PyTorch gradients, I make a copy of these gradients and add some noise to it. For each batch, I check the loss for the original gradients and I check the loss for the new gradients. I pick the gradients that gives me lower loss values. <strong>While I alter gradients, I do not wish to alter optimiser momentum parameters learnt via optimiser.step()</strong>. Can you let me know answers to the following two questions?</p>
<ol>
<li>
<p>How do I reference all gradients at once (instead of by layer name like <em>model.conv1.grad</em>) may be like a list comprehension and assign values?</p>
</li>
<li>
<p>What mistake am I doing in my code? Do I need to set gradients to zero somehow like <em>grad.zero_()</em>? If so, I think I need <em>requires_grad = True</em>. So, is deep copy of gradients a solution? If I do a deep copy of the model, how do I assign the original optimiser parameters to the new model?</p>
</li>
</ol>
<p><strong>P.S.:</strong> For reference, I have also included important function:</p>
<pre><code class="lang-auto">def train_epoch(eta, model, train_loader, criterion):
    model.train()

    running_loss = 0.0
    predictions = []
    ground_truth = []
    loss_den = 1
    
    start_time = time.time()
    optimiser = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
    
    for batch_idx, (data, target) in enumerate(train_loader):
        
        data = data.to(device)
        target = target.to(device)
    
        #previous model
        outputs = model(data.float())
        _, predicted = torch.max(outputs.data, 1)
        total_predictions = target.size(0)
        correct_predictions = (predicted == target).sum().item()
        acc = (correct_predictions/total_predictions)*100.0
        
        loss = criterion(outputs, target)
        loss.backward()
        optimiser.step()
        
        #convGrad is the set of old gradients
        conv1grad = model.conv1.weight.grad
        conv2grad = model.conv2.weight.grad
        conv3grad = model.conv3.weight.grad
                
        noisyGrad1 = eta * np.abs(conv1grad.detach().cpu().numpy())
        noisyGrad2 = eta * np.abs(conv2grad.detach().cpu().numpy())
        noisyGrad3 = eta * np.abs(conv3grad.detach().cpu().numpy())
        
        newGrad1 = conv1grad + torch.from_numpy(np.random.uniform(-noisyGrad1, noisyGrad1))
        newGrad2 = conv2grad + torch.from_numpy(np.random.uniform(-noisyGrad2, noisyGrad2))
        newGrad3 = conv3grad + torch.from_numpy(np.random.uniform(-noisyGrad3, noisyGrad3))
        
        model.conv1.weight.grad = nn.Parameter(torch.from_numpy(newGrad1.detach().numpy()).float())
        model.conv2.weight.grad = nn.Parameter(torch.from_numpy(newGrad2.detach().numpy()).float())
        model.conv3.weight.grad = nn.Parameter(torch.from_numpy(newGrad3.detach().numpy()).float())
        
        #The new loss value for the new gradients is computed
        outputs = model(data.float())
        _, predicted = torch.max(outputs.data, 1)
        total_predictions = target.size(0)
        correct_predictions = (predicted == target).sum().item()
        acc_new = (correct_predictions/total_predictions)*100.0
        
        loss_new = criterion(outputs, target)
        loss_den += 1

        #calculuating confusion matrix
        predictions += list(predicted.detach().cpu().numpy())
        ground_truth += list(target.detach().cpu().numpy())

        if loss_new.item() &gt; loss.item():
            model.conv1.weight.grad = conv1grad
            model.conv2.weight.grad = conv2grad
            model.conv3.weight.grad = conv3grad

            running_loss += loss.item()
        else:
            running_loss += loss_new.item()
        
    end_time = time.time()

    running_loss /= loss_den
    
    print('Training Loss: ', running_loss, 'Time: ',end_time - start_time, 's')
    
    return running_loss, model
</code></pre>

Prasad_Raghavendra: <p>I read my own question again (typed while I was having some other work; Sorry for that). My question is somewhat difficult to comprehend when seeing for the first time. So, let me ask questions one by one. First question:</p>
<p>How do I access all weights of a model? Currently, I am accessing with layer names like <em>model.conv1.weight.grad</em>. But, is there anything like <em>model.weight.grad</em> to access all gradients (setter and getter)?</p>
<p>To all PyTorch developers, thanks for the beautiful library and to those helping, thanks for your time.</p>

albanD: <p>Hi,</p>
<p>You will still have to access the gradients for each weight one by one but you can do that automatically with:</p>
<pre><code class="lang-python">for param in model.parameters():
    param.grad # Some something with it here</code></pre>

Prasad_Raghavendra: <p>Thank you so much for this. It answered my first question so well. Let me consolidate my code and ask the same question better.</p>
<p>My next and last question is how I can undo <em>loss.backward()</em> for a given batch. I wish to compute the autograd backprop gradients. But, I do not wish to apply those gradients to model computation graph. How could I do that?</p>
<p>The final objective is to get two sets of gradients for each batch (regular and noisy one) and apply the gradient set that produces lower loss values. How can I achieve that?</p>
<p>The cleaner code (thanks to <a class="mention" href="/u/alband">@albanD</a> ) is as follows:</p>
<pre><code class="lang-auto">def train_epoch(eta, model, train_loader, criterion):
    model.train()
    running_loss = 0.0; predictions = []; ground_truth = []; loss_den = 1
    
    start_time = time.time()
    optimiser = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)
    
    for batch_idx, (data, target) in enumerate(train_loader):
        
        data = data.to(device); target = target.to(device)
    
        #previous model
        outputs = model(data.float())        
        loss = criterion(outputs, target)
        loss.backward()
        optimiser.step()
        
        #convGrad is the set of old gradients
        oldGrad = list()
        for param in model.parameters():
            convGrad = param.grad
            oldGrad.append(convGrad)
            noisyGrad = eta * np.abs(convGrad.detach().cpu().numpy())
            newGrad = convGrad + torch.from_numpy(np.random.uniform(-noisyGrad, noisyGrad))
            param.grad = nn.Parameter(torch.from_numpy(newGrad.detach().numpy()).float())
        
        #The new loss value for the new gradients is computed
        outputs = model(data.float())
        loss_new = criterion(outputs, target)
        loss_den += 1

        if loss_new.item() &gt; loss.item():
            for paramIdx, param in enumerate(model.parameters()):
                param.grad = oldGrad[paramIdx]

            running_loss += loss.item()
        else:
            running_loss += loss_new.item()
        
    end_time = time.time()
    running_loss /= loss_den
    print('Training Loss: ', running_loss, 'Time: ',end_time - start_time, 's')
    
    return running_loss, model
</code></pre>

Prasad_Raghavendra: <p>To summarise my previous question:</p>
<ol>
<li>
<p>How do I compute backprop gradients for a given batch but, not apply those gradients to the model automatically?</p>
</li>
<li>
<p>How can I do optimiser.step() for the gradients I manually apply to the model?</p>
</li>
</ol>

albanD: <p>Hi,</p>
<ol>
<li>You can use <code>autograd.grad()</code> to get the value of the gradient as a list and not modify the <code>.grad</code> fields of the parameters.</li>
<li>You will need to update the <code>.grad</code> fields before calling the <code>optimizer.step()</code> function I’m afraid.</li>
</ol>
<p>Also I am fairly confused by this line: <code>param.grad = nn.Parameter(torch.from_numpy(newGrad.detach().numpy()).float())</code> why do you wrap the grad in a Parameter? And why do you send it to numpy?</p>

Prasad_Raghavendra: <p>Thank you so much for your time and helpful reply.</p>
<blockquote>
<p>Also I am fairly confused by this line: <code>param.grad = nn.Parameter(torch.from_numpy(newGrad.detach().numpy()).float())</code> why do you wrap the grad in a Parameter? And why do you send it to numpy?</p>
</blockquote>
<p>I tried reading it up before replying. I was going through <a href="http://deeplearning.cs.cmu.edu/F20/index.html" class="inline-onebox" rel="noopener nofollow ugc">11-785 Deep Learning</a> and I think PyTorch is similar. I am still confused by autograd (even after a week) if I have to manually apply gradients myself.</p>
<p>From what I know, <code>Parameter</code> wraps a numpy array as a tensor. Because, I want to select the gradients manually and apply at each iteration, my gradients are in numpy array (because I am more familiar with numpy as compared to PyTorch).</p>
<blockquote>
<p>You will need to update the <code>.grad</code> fields before calling the <code>optimizer.step()</code> function I’m afraid.</p>
</blockquote>
<p>Ok. But, how do I invoke <code>autograd.grad()</code> for a model (without applying gradients to the model)? I was seeing <a href="https://pytorch.org/docs/stable/autograd.html#torch.autograd.grad" class="inline-onebox" rel="noopener nofollow ugc">Automatic differentiation package - torch.autograd — PyTorch 2.1 documentation</a> . So, do I do <code>model.grad()</code>?</p>

albanD: <aside class="quote no-group" data-username="Prasad_Raghavendra" data-post="7" data-topic="116268">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/prasad_raghavendra/48/24424_2.png" class="avatar"> Prasad_Raghavendra:</div>
<blockquote>
<p><code>Parameter</code> wraps a numpy array as a tensor.</p>
</blockquote>
</aside>
<p>That is not true.<br>
To get a Tensor from a numpy array, you can do <code>t = torch.from_numpy(your_numpy_array)</code>.<br>
A <code>nn.Parameter</code> is a thin wrapper around Tensor that has special meaning in the <code>torch.nn</code> library. Namely it is a parameter of a Module and so will be returned by <code>mod.parameters()</code> call.</p>
<blockquote>
<p>But, how do I invoke <code>autograd.grad()</code></p>
</blockquote>
<p><code>grads = autograd.grad(loss, model.parameters())</code> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

Prasad_Raghavendra: <blockquote>
<p>Namely it is a parameter of a Module and so will be returned by <code>mod.parameters()</code> call.</p>
</blockquote>
<p>Thank you! Let me format the code and ask.</p>
<aside class="quote no-group" data-username="albanD" data-post="8" data-topic="116268">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar"> albanD:</div>
<blockquote>
<p><code>grads = autograd.grad(loss, model.parameters())</code> <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
</blockquote>
</aside>
<p>This is so helpful! Thank you <a class="mention" href="/u/alband">@albanD</a> .</p>

Prasad_Raghavendra: <p>Even after some 5 years, this is so useful, really beautiful answer, really beautiful library and (of course, a really beautiful question <img src="https://discuss.pytorch.org/images/emoji/apple/wink.png?v=12" title=":wink:" class="emoji" alt=":wink:" loading="lazy" width="20" height="20"> ).</p>
<p>A follow-up question. I am making sure I am using the grad function correctly. If I need to test out some learning rates and pick one that works. I am doing a “test” for that. Is this how it is supposed to be used?</p>
<pre><code class="lang-auto">def test_lr(lr, CNN_model, data, target, previous_loss, current_loss):
    output = CNN_model(data); loss = criterion(output, target)
    grad = torch.autograd.grad(loss, CNN_model.parameters())

    for parameter in CNN_model.parameters(): parameter.data -= lr * grad
    output = CNN_model(data); loss = criterion(output, target); future_loss = loss.item()
    test_result = False if future_loss &gt; current_loss or ((future_loss + previous_loss) / 2) &gt; current_loss else True

    return test_result
</code></pre>
<p>It throws this error: <code>TypeError: can't multiply sequence by non-int of type 'float'</code> (which I assume is because of scalar multiplication with <code>grad</code>)</p>

Prasad_Raghavendra: <p>Actually, I was able to resolve the <code>TypeError</code> using <code>for parameter, grad_value in zip(CNN_model.parameters(), grad): parameter.data -= lr * grad_value</code> (in part due to Meta AI in WhatsApp)</p>
<p>I have a very simple code which is about doing a binary search. I am not sure why <code>test_right_result</code> works but, not <code>test_left_result</code> (which is very strange)</p>
<p>The main functions:</p>
<pre><code class="lang-auto">def init_binary_search_tree():
    """
    I hope this gets integrated into PyTorch torch optim. This dict version is bad. I am making it simple as of now.
    """
    d = dict()

    d[-2] = {"right": -1, "left": -3}
    d[-6] = {"right": -5, "left": -7}
    d[-10] = {"right": -9, "left": -11}
    d[-14] = {"right": -13, "left": -15}

    d[-4] = {"right": -2, "left": -6}
    d[-12] = {"right": -10, "left": -14}

    d[-8] = {"right": -4, "left": -12}
    return d

def test_lr(lr, CNN_model, data, target, previous_loss, current_loss):
    output = CNN_model(data); loss = criterion(output, target)
    grad = torch.autograd.grad(loss, CNN_model.parameters())

    for parameter, grad_value in zip(CNN_model.parameters(), grad): parameter.data -= lr * grad_value
    output = CNN_model(data); loss = criterion(output, target); future_loss = loss.item()
    test_result = False if future_loss &gt; current_loss or ((future_loss + previous_loss) / 2) &gt; current_loss else True

    return test_result

def reset(d, data, target, CNN_model, previous_loss, current_loss):
    node = -8; lr = 2 ** -16

    for _ in range(3):
        # Check right node first; Check left node next; If none of those work, "stay" at the current node
        lr_right = 2 ** d[node]["right"]; lr_left = 2 ** d[node]["left"]
        test_right_result = test_lr(lr_right, CNN_model, data, target, previous_loss, current_loss)

        if test_right_result == False:
            test_left_result = test_lr(lr_left, CNN_model, data, target, previous_loss, current_loss)
            print(lr_left, test_left_result)
            if test_left_result == False:
                lr = 2 ** node; break
            else:
                lr = lr_left; node = d[node]["left"]
        else:
            lr = lr_right; node = d[node]["right"]

    return lr, CNN_model
</code></pre>