Pippy I can't see backward pass

nomaue: <p>Hi!</p>
<p>I’m using pippy package and I experimented the following code.</p>
<p><a href="https://github.com/pytorch/tau/blob/main/examples/resnet/pippy_resnet.py" class="onebox" target="_blank" rel="noopener nofollow ugc">https://github.com/pytorch/tau/blob/main/examples/resnet/pippy_resnet.py</a></p>
<p>I only changed dataset from cifar10 to imagenet, and world size from 5 to 4(annotate_split_points block too.) because I only have 4 gpus.<br>
But, I can’t see backward pass.<br>
this is the result of <strong>print(pipe)</strong></p>
<pre><code class="lang-auto">def forward(self, input, target):
    input_1 = input
    submod_0 = self.submod_0(input_1);  input_1 = None
    submod_1 = self.submod_1(submod_0);  submod_0 = None
    submod_2 = self.submod_2(submod_1, target);  submod_1 = target = None
    getitem = submod_2[0]
    getitem_1 = submod_2[1];  submod_2 = None
    sync_barrier = pippy_backward_sync_barrier((getitem, getitem_1), [], None);  getitem = getitem_1 = None
    return sync_barrier
    
</code></pre>
<p>there’s no stage_backward.<br>
and there’s only forward pass stage in tensorbard as you can see…<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/4/64644b79b5644a96acec720601ae614d0583fe0b.png" data-download-href="https://discuss.pytorch.org/uploads/default/64644b79b5644a96acec720601ae614d0583fe0b" title="resnet pp no dp"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/64644b79b5644a96acec720601ae614d0583fe0b_2_690x317.png" alt="resnet pp no dp" data-base62-sha1="ek6CUWQRSAWDF3ewDdZUB7m6qRZ" width="690" height="317" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/64644b79b5644a96acec720601ae614d0583fe0b_2_690x317.png, https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/64644b79b5644a96acec720601ae614d0583fe0b_2_1035x475.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/6/4/64644b79b5644a96acec720601ae614d0583fe0b_2_1380x634.png 2x" data-dominant-color="F3F6F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">resnet pp no dp</span><span class="informations">2224×1023 98.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>how can i get the stage backward?<br>
Thank you.</p>

fduwjj: <p>cc: <a class="mention" href="/u/kwen2501">@kwen2501</a> on PiPPy example question.</p>

nomaue: <p>edited// this reply was my mistake</p>

nomaue: <p>I changed LossWrapper block to</p>
<pre><code class="lang-auto">
    class OutputLossWrapper(LossWrapper):
        def forward(self, x, target):
            return self.loss_fn(self.module(x), target) 

    wrapper = OutputLossWrapper(model, cross_entropy)

</code></pre>
<p>.<br>
the original version from github example I mentioned is</p>
<pre><code class="lang-auto">    class OutputLossWrapper(LossWrapper):
        def __init__(self, module, loss_fn):
            super().__init__(module, loss_fn)

        def forward(self, input, target):
            output = self.module(input)
            return output, self.loss_fn(output, target)

    wrapper = OutputLossWrapper(model, cross_entropy)
</code></pre>
<p>.<br>
.<br>
.<br>
and now I got stage_backward like this</p>
<pre><code class="lang-auto">def forward(self, x, target):
    submod_0 = self.submod_0(x)
    submod_1 = self.submod_1(submod_0)
    submod_2 = self.submod_2(submod_1, target)
    stage_backward = pippy_backward_stage_backward(stage_output = (submod_2,), output_grads = (None,), input_values = [submod_1, target], outputs_with_grads_idxs = [0], stage_info = 'stage_backward for stage %submod_2 : [#users=2] = call_module[target=submod_2](args = (%submod_1, %target), kwargs = {})');  target = None
    getitem = stage_backward[0]
    getitem_1 = stage_backward[1];  stage_backward = None
    getitem_2 = getitem[0]
    getitem_3 = getitem[1];  getitem = None
    stage_backward_1 = pippy_backward_stage_backward(stage_output = (submod_1,), output_grads = (getitem_2,), input_values = [submod_0], outputs_with_grads_idxs = [0], stage_info = 'stage_backward_1 for stage %submod_1 : [#users=3] = call_module[target=submod_1](args = (%submod_0,), kwargs = {})');  submod_1 = getitem_2 = None
    getitem_4 = stage_backward_1[0]
    getitem_5 = stage_backward_1[1];  stage_backward_1 = None
    getitem_6 = getitem_4[0];  getitem_4 = None
    stage_backward_2 = pippy_backward_stage_backward(stage_output = (submod_0,), output_grads = (getitem_6,), input_values = [x], outputs_with_grads_idxs = [0], stage_info = 'stage_backward_2 for stage %submod_0 : [#users=3] = call_module[target=submod_0](args = (%x,), kwargs = {})');  submod_0 = getitem_6 = x = None
    getitem_7 = stage_backward_2[0]
    getitem_8 = stage_backward_2[1];  stage_backward_2 = None
    getitem_9 = getitem_7[0]
    sync_barrier = pippy_backward_sync_barrier(submod_2, [getitem_1, getitem_5, getitem_8], getitem_7);  submod_2 = getitem_1 = getitem_5 = getitem_8 = getitem_7 = None
    return sync_barrier
 
</code></pre>
<p>but it doesn’t seem to be trained well.<br>
and I printed loss value(=pipe_driver(x, target)) , It is greater than 1.</p>
<p>Why doesn’t the original OutputLossWrapper make backward stage?</p>

kwen2501: <p>Hi <a class="mention" href="/u/nomaue">@nomaue</a> thanks for reporting this issue and glad you found a workaround.<br>
We will investigate why returning two outputs stops PiPPy from generating the backward pass and report back here.</p>

nomaue: <p>Thank you <a class="mention" href="/u/kwen2501">@kwen2501</a>!<br>
I have one more question.<br>
I also experimented the following code, but it seems not to work for pipelining.</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/tau/blob/main/examples/ddp2pipe/ddp2pipe.py">
  <header class="source">

      <a href="https://github.com/pytorch/tau/blob/main/examples/ddp2pipe/ddp2pipe.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/tau/blob/main/examples/ddp2pipe/ddp2pipe.py" target="_blank" rel="noopener nofollow ugc">pytorch/tau/blob/main/examples/ddp2pipe/ddp2pipe.py</a></h4>


      <pre><code class="lang-py">import argparse
# import logging
import os
import socket

import torch
import torch.distributed
import torch.distributed.rpc as rpc
import torch.multiprocessing as mp
import torch.nn.functional as F
from torch import nn, optim
from torch.nn.parallel import DistributedDataParallel as DDP
from torch.utils.data import DistributedSampler
from torchvision import datasets
from torchvision.transforms import transforms
from tqdm import tqdm

from pippy import Pipe, PipelineDriverFillDrain, annotate_split_points, PipeSplitWrapper
from pippy.microbatch import TensorChunkSpec, CustomReducer
from pippy.utils import tp_transports
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/tau/blob/main/examples/ddp2pipe/ddp2pipe.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>when I set</p>
<pre><code class="lang-auto">DIMS = [28 * 28, 300, 100, 10]
DP_LAYERS = 2
PP_LAYERS = 1
#nnode=1, nproc_per_node=2
</code></pre>
<p>It works well for Dataparallel,<br>
but when I set like below,</p>
<pre><code class="lang-auto">DIMS = [28 * 28, 300, 100, 10]
DP_LAYERS = 1
PP_LAYERS = 2
#nnode=1, nproc_per_node=2

</code></pre>
<p>or</p>
<pre><code class="lang-auto">DIMS = [28 * 28, 500, 250, 100, 50, 25, 10]
DP_LAYERS = 2
PP_LAYERS = 4
#nnode=2, nproc_per_node=4
</code></pre>
<p>It doesn’t work at all. could this problem be related to the issue I suggested at first? or did I set somethin incorreclty? If I need to post a new topic for this, I’ll do that.<br>
Thank you.</p>

kwen2501: <p>Hi <a class="mention" href="/u/nomaue">@nomaue</a>, I just came back from PTO.<br>
Regarding the issue of PiPPy failing to generate the backward pass, I submitted a fix here:</p><aside class="onebox githubpullrequest" data-onebox-src="https://github.com/pytorch/tau/pull/738">
  <header class="source">

      <a href="https://github.com/pytorch/tau/pull/738" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/tau</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">



    <div class="github-icon-container" title="Pull Request">
      <svg width="60" height="60" class="github-icon" viewbox="0 0 12 16" aria-hidden="true"><path fill-rule="evenodd" d="M11 11.28V5c-.03-.78-.34-1.47-.94-2.06C9.46 2.35 8.78 2.03 8 2H7V0L4 3l3 3V4h1c.27.02.48.11.69.31.21.2.3.42.31.69v6.28A1.993 1.993 0 0 0 10 15a1.993 1.993 0 0 0 1-3.72zm-1 2.92c-.66 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2zM4 3c0-1.11-.89-2-2-2a1.993 1.993 0 0 0-1 3.72v6.56A1.993 1.993 0 0 0 2 15a1.993 1.993 0 0 0 1-3.72V4.72c.59-.34 1-.98 1-1.72zm-.8 10c0 .66-.55 1.2-1.2 1.2-.65 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2zM2 4.2C1.34 4.2.8 3.65.8 3c0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2z"></path></svg>
    </div>

  <div class="github-info-container">



      <h4>
        <a href="https://github.com/pytorch/tau/pull/738" target="_blank" rel="noopener nofollow ugc">Fix issue that PiPPy does not generate backward pass when there are multiple output values</a>
      </h4>

    <div class="branches">
      <code>pytorch:main</code> ← <code>pytorch:fix_find_loss_from_wrapper</code>
    </div>

      <div class="github-info">
        <div class="date">
          opened <span class="discourse-local-date" data-format="ll" data-date="2023-02-07" data-time="20:46:03" data-timezone="UTC">08:46PM - 07 Feb 23 UTC</span>
        </div>

        <div class="user">
          <a href="https://github.com/kwen2501" target="_blank" rel="noopener nofollow ugc">
            <img alt="kwen2501" src="https://avatars.githubusercontent.com/u/6676466?v=4" class="onebox-avatar-inline" width="20" height="20">
            kwen2501
          </a>
        </div>

        <div class="lines" title="1 commits changed 2 files with 8 additions and 3 deletions">
          <a href="https://github.com/pytorch/tau/pull/738/files" target="_blank" rel="noopener nofollow ugc">
            <span class="added">+8</span>
            <span class="removed">-3</span>
          </a>
        </div>
      </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Issue was reported by user when trying the ResNet example:
[Pippy I can’t see b<span class="show-more-container"><a href="https://github.com/pytorch/tau/pull/738" target="_blank" rel="noopener nofollow ugc" class="show-more">…</a></span><span class="excerpt hidden">ackward pass](https://discuss.pytorch.org/t/pippy-i-cant-see-backward-pass/170630)

### Explanation of the Issue:
When the output is in tuple form and there other values than loss in the tuple, e.g. `return output, loss`, PiPPy could not figure out which field is the loss, and hence did not generate the backward pass.

### Fix to the ResNet example:
The return needs to be in a form of dict, e.g. `return {"output": output, "loss": loss}`, so that PiPPy can automatically locate the loss field by looking at the keyword.

### Fix to PiPPy's IR:
We cannot assume that there is only one output value (loss) when the model is of the general `LossWrapper` class. Such assumption is only valid for the `TrivialLossWrapper` class.</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
Thanks for reporting this issue!</p>

kwen2501: <p>The issue in the ddp2pipe example is a separate issue.<br>
The model in the ddp2pipe example contains two parts: a tradition DDP part followed by a pipeline. And a special connection is built to connect those two parts.<br>
It is not very stable for now as it only works for the parameters specified in the example. It may also work for CPU only due to a hang in GPU mode.</p>

nomaue: <p>Many thanks for you reply <a class="mention" href="/u/kwen2501">@kwen2501</a>. I’ll try running the code.</p>

nomaue: <p>Hi <a class="mention" href="/u/kwen2501">@kwen2501</a> . so, Is ‘PP+DP’ unavailable on single or multi-node so for? for both any cnn model and transformer?</p>

kwen2501: <p>Hi,<br>
PP + DP is available today through PiPPy’s <code>init_data_parallel</code> API.</p>
<p><strong>Documentation</strong>:</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/pytorch/tau#pippy--data-parallelism">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/pytorch/tau#pippy--data-parallelism" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/bc21e16b74883dec8fae405c29b02ffef208faf0497fd7f3f0573953b621655e/pytorch/tau" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/pytorch/tau#pippy--data-parallelism" target="_blank" rel="noopener nofollow ugc">GitHub - pytorch/tau: Pipeline Parallelism for PyTorch</a></h3>

  <p>Pipeline Parallelism for PyTorch. Contribute to pytorch/tau development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p><strong>Example</strong>:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/tau/blob/main/test/local_test_ddp.py">
  <header class="source">

      <a href="https://github.com/pytorch/tau/blob/main/test/local_test_ddp.py" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/tau/blob/main/test/local_test_ddp.py" target="_blank" rel="noopener nofollow ugc">pytorch/tau/blob/main/test/local_test_ddp.py</a></h4>


      <pre><code class="lang-py"># Copyright (c) Meta Platforms, Inc. and affiliates
import argparse
import copy
import os
import unittest

import torch
import torch.distributed.rpc as rpc

import pippy.fx
from pippy import run_pippy
from pippy.IR import (
    MultiUseParameterConfig,
    Pipe,
    TrivialLossWrapper,
    pipe_split,
)
from pippy.PipelineDriver import (
    PipelineDriverFillDrain,
    PipelineDriver1F1B,
</code></pre>



  This file has been truncated. <a href="https://github.com/pytorch/tau/blob/main/test/local_test_ddp.py" target="_blank" rel="noopener nofollow ugc">show original</a>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>