Inference on multi GPU

username: Milad_Yazdani
cooked: <p>Hi, I have a sizeable pre-trained model and I want to get inference on multiple GPU from it(I don’t want to train it).so is there any way for that?<br>
In summary, I want model-parallelism. and if there is a way, how is it done?</p>

username: pritamdamania87
cooked: <p><a class="mention" href="/u/milad_yazdani">@Milad_Yazdani</a> There are multiple options depending on the type of model parallelism you want. There is PyTorch FSDP: <a href="https://pytorch.org/docs/stable/fsdp.html" class="inline-onebox" rel="noopener nofollow ugc">FullyShardedDataParallel — PyTorch 1.11.0 documentation</a> which is ZeRO3 style for large models. There is very recent Tensor Parallelism support (see this example: <a href="https://github.com/pytorch/examples/tree/main/distributed/sharded_tensor" class="inline-onebox" rel="noopener nofollow ugc">examples/distributed/sharded_tensor at main · pytorch/examples · GitHub</a>) and there is pipeline parallelism support too: <a href="https://github.com/pytorch/PiPPy" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/PiPPy: Pipeline Parallelism for PyTorch</a></p>
<p>If you could share more details about your model and setup we can help in proposing what might be the best fit here:</p>
<ol>
<li>How big is the model (number of parameters) and how many GPUs do you want to use?</li>
<li>Do you want to split the model across multiple GPUs on a single host or is the model large enough that it needs to be split across multiple hosts?</li>
<li>Since this is GPU inference, I’m assuming you want to optimize for latency?</li>
</ol>

username: pritamdamania87
cooked: <p>cc <a class="mention" href="/u/mrshenli">@mrshenli</a> Regarding distributed inference.</p>

username: kwen2501
cooked: <p><a href="https://github.com/pytorch/tau" rel="noopener nofollow ugc">PiPPy</a> (Pipeline Parallelism for PyTorch) supports distributed inference.</p>
<p>PiPPy can split <strong>pre-trained</strong> models into pipeline stages and distribute them onto multiple GPUs or even multiple hosts. It also supports distributed, per-stage materialization if the model does not fit in the memory of a single GPU. When you have multiple <strong>microbatches</strong> to inference, pipeline parallelism is achieved.</p>
<p>Here is an inference example of large T5 models using PiPPy, with detailed user guide:</p><aside class="onebox githubpullrequest" data-onebox-src="https://github.com/pytorch/tau/pull/645">
  <header class="source">

      <a href="https://github.com/pytorch/tau/pull/645" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/tau</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">



    <div class="github-icon-container" title="Pull Request">
      <svg width="60" height="60" class="github-icon" viewbox="0 0 12 16" aria-hidden="true"><path fill-rule="evenodd" d="M11 11.28V5c-.03-.78-.34-1.47-.94-2.06C9.46 2.35 8.78 2.03 8 2H7V0L4 3l3 3V4h1c.27.02.48.11.69.31.21.2.3.42.31.69v6.28A1.993 1.993 0 0 0 10 15a1.993 1.993 0 0 0 1-3.72zm-1 2.92c-.66 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2zM4 3c0-1.11-.89-2-2-2a1.993 1.993 0 0 0-1 3.72v6.56A1.993 1.993 0 0 0 2 15a1.993 1.993 0 0 0 1-3.72V4.72c.59-.34 1-.98 1-1.72zm-.8 10c0 .66-.55 1.2-1.2 1.2-.65 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2zM2 4.2C1.34 4.2.8 3.65.8 3c0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2z"></path></svg>
    </div>

  <div class="github-info-container">



      <h4>
        <a href="https://github.com/pytorch/tau/pull/645" target="_blank" rel="noopener nofollow ugc">[WIP] Inference examples</a>
      </h4>

    <div class="branches">
      <code>pytorch:main</code> ← <code>pytorch:inference_examples</code>
    </div>

      <div class="github-info">
        <div class="date">
          opened <span class="discourse-local-date" data-format="ll" data-date="2022-11-23" data-time="00:16:51" data-timezone="UTC">12:16AM - 23 Nov 22 UTC</span>
        </div>

        <div class="user">
          <a href="https://github.com/HamidShojanazeri" target="_blank" rel="noopener nofollow ugc">
            <img alt="HamidShojanazeri" src="https://avatars.githubusercontent.com/u/9162336?v=4" class="onebox-avatar-inline" width="20" height="20">
            HamidShojanazeri
          </a>
        </div>

        <div class="lines" title="15 commits changed 3 files with 409 additions and 0 deletions">
          <a href="https://github.com/pytorch/tau/pull/645/files" target="_blank" rel="noopener nofollow ugc">
            <span class="added">+409</span>
            <span class="removed">-0</span>
          </a>
        </div>
      </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">WIP to make a official example and recipe for distributed inference with pippy. <span class="show-more-container"><a href="https://github.com/pytorch/tau/pull/645" target="_blank" rel="noopener nofollow ugc" class="show-more">…</a></span><span class="excerpt hidden">

Goals

- To make a recipe for distributed inference
- Identify areas to simplify and improve</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>