Why losses combined as loss_dict = model(images, targets) losses = sum(loss for loss in loss_dict.values()) optimizer.zero_grad() losses.backward()

username: user29
cooked: <p>Hello everyone!<br>
Why losses combined as<br>
<em>loss_dict = model(images, targets) losses = sum(loss for loss in loss_dict.values()) optimizer.zero_grad() losses.backward()</em> but never as<br>
<em>losses = sum(loss_dict.values())</em> ? Is it the same ?</p>

username: my3bikaht
cooked: <p>You have very specific model that returns losses instead of predictions and it also returns them as a dictionary. Usually models return predictions, then loss functions return loss(es) as tensor(s) and you can access it’s value via .item()</p>

username: user29
cooked: <p>Thank you for answer!</p>
<p>Like we can see here <a href="https://github.com/pytorch/vision/blob/59ec1dfd550652a493cb99d5704dcddae832a204/references/detection/engine.py" class="inline-onebox" rel="noopener nofollow ugc">vision/engine.py at 59ec1dfd550652a493cb99d5704dcddae832a204 · pytorch/vision · GitHub</a> all models in torchvision in train state return dictionary of losses (of course in state eval they return prediction).  I think it is common way because it incapsulate losses in model that very convenient.</p>
<p>But anyway, when we combine losses it has a lot of subtleties because we combine graphs with gradient not just values. Here more details <a href="https://stackoverflow.com/questions/53994625/how-can-i-process-multi-loss-in-pytorch/53997034#53997034" class="inline-onebox" rel="noopener nofollow ugc">python - How can i process multi loss in pytorch? - Stack Overflow</a></p>
<p>I guess that exist reason related to autograd or something else  why in all place losses combinated via python сomprehension  sum(loss for loss in loss_dict_reduced.values()) not just sum(loss_dict_reduced.values())</p>
<p>One man told me that it related with torchscript that it not supports sum of dictionary values by direct. But he not sure.</p>

username: my3bikaht
cooked: <p>Ah, I see. I think this is because dict.values() returned as itervalues iterator, and [loss for loss in itervalues()] is converting iterator to a list.</p>

username: user29
cooked: <p>Yes, that is truth. [loss for loss in itervalues()] is converting iterator to a list. But question was why we should convert iterator to a list and than sum it sum(loss for loss in loss_dict.values())?  Why we cannot sum iterator directly sum(loss_dict.values()) ? What is different ?</p>

username: my3bikaht
cooked: <p>Right. That code was written 3 years ago, I guess torch.sum was not able to sum over iterator in early 2019.</p>

username: user29
cooked: <p>Maybe.<br>
But in code like this</p>
<pre><code class="lang-auto">  with torch.cuda.amp.autocast(enabled=scaler is not None):
            loss_dict = model(images, targets)
            losses = sum(loss for loss in loss_dict.values())

        # reduce losses over all GPUs for logging purposes
        loss_dict_reduced = utils.reduce_dict(loss_dict)
        losses_reduced = sum(loss for loss in loss_dict_reduced.values())

        loss_value = losses_reduced.item()

</code></pre>
<p>or this</p>
<pre><code class="lang-auto">  for i, data in enumerate(train_dataloader):
        optimizer.zero_grad()
        images, targets = data[0], data[1]
        images = list(image.to(device) for image in images)
        targets = [{k: v.to(device) for k, v in t.items()} for t in targets]
        loss_dict = model(images, targets)
        loss = sum(loss for loss in loss_dict.values())
        running_loss += loss.item()
        loss.backward()
        optimizer.step()
</code></pre>
<p>we cannot use torch.sum at all because it sum tensor whereas loss_dict.values() is list.</p>
<p>If we use google we see a lot of examples where used variant <code>loss = sum(loss for loss in loss_dict.values())</code> and never variant <code>sum(loss_dict.values())</code>. <a href="https://www.google.com/search?q=loss_dict.values+pytorch&amp;newwindow=1&amp;ei=E_e4YfDdJaHnrgTV8bT4Dg&amp;start=0&amp;sa=N&amp;ved=2ahUKEwiws9zAieT0AhWhs4sKHdU4De84ChDx0wN6BAgBEDk&amp;cshid=1639512048508418&amp;biw=1422&amp;bih=747&amp;dpr=1.75" class="inline-onebox" rel="noopener nofollow ugc">loss_dict.values pytorch - Google Search</a></p>
<p>I found another variant here <a href="https://isrc.iscas.ac.cn/gitlab/research/domain-adaption/-/commit/7fb5ee486e519de46175a44f6f67455334d4a549" rel="noopener nofollow ugc">https://isrc.iscas.ac.cn/gitlab/research/domain-adaption/-/commit/7fb5ee486e519de46175a44f6f67455334d4a549</a> but they do conversion too</p>
<pre><code class="lang-auto">        loss_dict, _ = model(images, img_metas, targets)
        losses = sum(list(loss_dict.values()))

        loss_dict_reduced = dist_utils.reduce_dict(loss_dict)
</code></pre>
<p>Maybe it only code style, maybe if we will use in our project variant without conversion <code>sum(loss_dict.values())</code> it will not lead to unwanted effects (like errors) but I not sure</p>

username: user29
cooked: <p>Right this the same. In this repository a lot of example <a href="https://github.com/facebookresearch/detectron2/search?q=loss_dict&amp;type=code" class="inline-onebox" rel="noopener nofollow ugc">Search · loss_dict · GitHub</a></p>