Manually setting a loss value

snxeno: <p>So here’s what I am trying to do:</p>
<p>I have an original model_0 and N copies of it model_{1,…,N} that are trained on different partitions of the data.<br>
I want to compute losses for model 1 through N, then use their summed losses as the loss for model_0.</p>
<p>What I do now is compute the sum of losses in a variable loss_total<br>
then I make a forward pass through model_0, and manually change the data variable to the wanted loss value</p>
<pre><code class="lang-auto">total_losses = #sum of losses from N copies
optimizer_0 = torch.optim.Adam(model_0.parameters(), lr=lr)

model_loss = loss(model_0(x_random), y_random) # make a forward pass through any data

model_loss.data = total_losses # manually change the loss

optimizer_0.zero_grad()
model_loss.backward()
optimizer_0.step()
</code></pre>
<p>I am not  sure that this is the way to go about it, so I appreciate any expert opinion on the matter.</p>
<p>Thanks,</p>

albanD: <p>Hi,</p>
<p>First of all, you should not use <code>.data</code> anymore.<br>
You can use <code>.detach()</code> or <code>torch.no_grad()</code> to replace it.</p>
<p>What is the behavior you expect for the gradients?<br>
Should they be the sum of the gradients for each models? Or should it be only for the first one with the value of the loss changed (so not the true gradients of that model but something else)?</p>

snxeno: <p>It should be the gradient of model_0 with the loss value changed.<br>
So it is equivalent of wanting to do gradient updates on the original model using a loss value that I manually enter.</p>

albanD: <p>Ok, can you write down mathematically what you want to do here please?<br>
Or write down the values you expect for a very simple example where your model+loss just computes <code>(2 * input).sum()</code></p>

snxeno: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/b/db82eec59226452a53ae7973d8d1018404c7a746.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/db82eec59226452a53ae7973d8d1018404c7a746" title="IMG_20191118_203351__01"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db82eec59226452a53ae7973d8d1018404c7a746_2_690x248.jpeg" alt="IMG_20191118_203351__01" data-base62-sha1="vjT5ssy9gVhzJQcqbXDbJpbRrxQ" width="690" height="248" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db82eec59226452a53ae7973d8d1018404c7a746_2_690x248.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db82eec59226452a53ae7973d8d1018404c7a746_2_1035x372.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db82eec59226452a53ae7973d8d1018404c7a746_2_1380x496.jpeg 2x" data-dominant-color="A19690"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">IMG_20191118_203351__01</span><span class="informations">3784×1364 1020 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I want to train N copies of the model on different data. After that, I want to use their summed losses as the loss for the original model (parameter index 0 in the image) and update the original model’s parameters based on that summed loss.</p>
<p>Let me know if that cleared it up. I am implementing the adaption step from the MAML algorithm.</p>
<p>Thank you,</p>

albanD: <p>But <code>nabla_theta_0 L(f_theta_i) = 0 \forall i != 0</code>. So you can simplify your expression to remove the sum and just keep <code>i=0</code>. Is that expected?</p>

snxeno: <p>Oh I think I understand the confusion.<br>
Every model that’s  <code>i != 0</code> has been trained on different data and each have a loss &gt; 0.<br>
Then I want the original model to treat that loss as its own, so the gradient of the loss wouldn’t be 0. I want to trick the model into thinking that it acquired the loss value on its own, and then do sgd on it.</p>

albanD: <p>Ok,<br>
And all the models have the same weights right?</p>

snxeno: <p>This is an example of how it should work:</p>
<pre><code class="lang-auto">model_0 = create_model()
summed_losses = 0
for _ in range(epochs):
  models_array = [deepcopy(model) for _ in range N] # get N copies of the main model
  for i in range(N):
    local_optimizer = optimizer(models_array[i].parameters(), lr)
    local_loss = loss(models_array[i](data[i]), labels[i]) 
    #each model is trained on different data

    local_optimizer.zero_grad()
    local_loss.backward()
    local_optimizer.step()

    with torch.no_grad():    
      summed_losses += loss(models_array[i](eval_data[i]), eval_labels[i]) #collect sum of losses for the main model

  main_optimizer = optimizer(model_0.parameters(), lr)
  main_loss = summed_losses #this is the part that I want to figure out. How do I assign summed_losses as the loss for my main model
  
  main_optimizer.zero_grad()
  main_loss.backward() #trick the main model to use the summed loss as its own
  main_optimizer.step() 
</code></pre>
<p>so at every iteration, I initiate N copies of the main model, then they train on different data and update their weights independently. Then I evaluate those N models and get a summation of their losses, and use that loss as the main model’s loss.</p>

albanD: <p>Sorry, this does not really make sense to me.<br>
Your models i&gt;0 have different weights than the weights for the one with i=0. So the gradients of the i&gt;0 models wrt w_0 are <code>0</code>.<br>
Given your torch.no_grad, I guess that you don’t plan on backpropagating through the weight updates.</p>

vainaijr: <p>I found one implementation of MAML in PyTorch here,<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/wyharveychen/CloserLookFewShot/blob/master/methods/maml.py" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/wyharveychen/CloserLookFewShot/blob/master/methods/maml.py" target="_blank" rel="nofollow noopener">wyharveychen/CloserLookFewShot/blob/master/methods/maml.py</a></h4>
<pre><code class="lang-py"># This code is modified from https://github.com/dragen1860/MAML-Pytorch and https://github.com/katerakelly/pytorch-maml 

import backbone
import torch
import torch.nn as nn
from torch.autograd import Variable
import numpy as np
import torch.nn.functional as F
from methods.meta_template import MetaTemplate

class MAML(MetaTemplate):
    def __init__(self, model_func,  n_way, n_support, approx = False):
        super(MAML, self).__init__( model_func,  n_way, n_support, change_way = False)

        self.loss_fn = nn.CrossEntropyLoss()
        self.classifier = backbone.Linear_fw(self.feat_dim, n_way)
        self.classifier.bias.data.fill_(0)
        
        self.n_task     = 4
        self.task_update_num = 5
</code></pre>

  This file has been truncated. <a href="https://github.com/wyharveychen/CloserLookFewShot/blob/master/methods/maml.py" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>they use fast weights for each layer in model, and do</p>
<pre><code class="lang-auto">loss_q = torch.stack(loss_all).sum(0)
loss_q.backward()
</code></pre>
<p>for summed loss</p>

snxeno: <p>I apologize for the confusion and I thank you for your patience with me.</p>
<p>The algorithm doesn’t really make sense outside of the few-shot learning framework.</p>
<p>I spent some time looking through the code <a class="mention" href="/u/vainaijr">@vainaijr</a> posted and how the algorithm should work and I think this is what it should do:</p>
<p>Models 1 through N compute losses for tasks 1 through N, then model_0 uses that to get its parameters to a place where the optimal weights for any of those tasks are a few gradient steps away from each of the tasks 1 through N.<br>
Models 1 through N have now accomplished their use, so we don’t need them for this iteration anymore.</p>
<p>In steps format:</p>
<ul>
<li>make N copies of model_0 (namely model 1 thorough N)</li>
<li>use models 1 through N to train them on different tasks and collect the losses</li>
<li>pretend that the summed loss is the loss for model_0</li>
<li>pass the same data models 1 through N trained on to model_0 and compute the loss on them, but manually change that loss so it’s equal to the summed loss</li>
<li>discard models 1 through N</li>
</ul>
<p>This happens for an <code>epochs</code> number of times, so the summed losses at each iteration work as an indicator for the main model as to how far it is from that good initialization place in the parameters space where most different tasks are a few gradient steps away.</p>
<p>That way model_0 can update its own weights based on the summed losses from models 1 through N.</p>
<p>My main question, which I think I didn’t ask very well, was how should I update the loss manually while not change the computational graph. Which I did using this way:</p>
<pre><code class="lang-auto">#data.x are all the data used to train models 1 through N. They are disjoint by nature.
model_loss = loss(model(data.x), data.y) 
            
with torch.no_grad():
 model_loss.set_(torch.Tensor([summed_losses]).to(device)[0]) 

</code></pre>
<p>Which is working for now. At least I think so <img src="https://discuss.pytorch.org/images/emoji/apple/man_shrugging.png?v=9" title=":man_shrugging:" class="emoji" alt=":man_shrugging:"></p>

albanD: <p>This is definitely the right way to change the value of the Tensor without changing the graph.</p>
<p>Just to be sure we understand each other. Here is a code sample that shows what this does (most likely nothing).</p>
<pre><code class="lang-python">import torch


# The loss is given by the sum operation.
# The backward of this op does not use the output value
def my_fn(x, modify_res=False):
    loss = (2*x).sum(0, keepdim=True)
    if modify_res:
        with torch.no_grad():
            loss[0] = 10
    return loss

inp = torch.rand(1, requires_grad=True)

my_fn(inp).backward()
print("base")
print(inp.grad)

inp.grad.zero_()

my_fn(inp, modify_res=True).backward()
print("modified")
print(inp.grad)


</code></pre>

Waterking: <p>Hello,<br>
I met the same problem.<br>
I wanna ask what my_fn and inp and modify_res means here, and why is loss[0] = 10?<br>
If I want to use loss calculated from a network to update another network, what should I do?<br>
Thanks in advance.</p>