Torch distributed launch & Flask Api

username: pemfir
cooked: <p>We have built an inference pipelines that take advantage of multiple GPU cores . We pass say a single image to <code>inference.py</code> using a shell script and it will return some results back. we have a shell script that contains the following:</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=1,2 python3 -m torch.distributed.launch --master_port 9800 --nproc_per_node=2 inference.py
</code></pre>
<p>in the <code>inference.py</code> we have</p>
<pre><code class="lang-auto">dist.init_process_group(backend="nccl", init_method="env://")
</code></pre>
<p>the code loads separate models into each GPU cores and runs them and then exits. Now we want to convert this script into a flask API.  This way, users can pass images to the inference.py from a front end web UI using a post command. So we made the following changes to the <code>inference.py</code></p>
<pre><code class="lang-auto">@app.route("/URL", methods=["POST"])
def search_engine():
    if request.method == "POST": 
        result = run_multi_GPU_code(request)
        return jsonify(result)


if __name__ == "__main__":

    dist.init_process_group(backend="nccl", init_method="env://")
    app.run(port=8115, host="0.0.0.0", debug=True)
</code></pre>
<p>and i get</p>
<pre><code class="lang-auto">  File "inference.py", line 55, in &lt;module&gt;
    dist.init_process_group(backend="nccl", init_method="env://")
  File "/usr/local/lib64/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 500, in init_process_group
    store, rank, world_size = next(rendezvous_iterator)
  File "/usr/local/lib64/python3.8/site-packages/torch/distributed/rendezvous.py", line 190, in _env_rendezvous_handler
    store = TCPStore(master_addr, master_port, world_size, start_daemon, timeout)
RuntimeError: Address already in use
</code></pre>
<p>Any help is appreciated.</p>

username: pritamdamania87
cooked: <aside class="quote no-group" data-username="pemfir" data-post="1" data-topic="145855">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pemfir/48/43803_2.png" class="avatar"> pemfir:</div>
<blockquote>
<pre><code class="lang-auto">RuntimeError: Address already in use
</code></pre>
</blockquote>
</aside>
<p>This typically means the port (I guess 8115) is already in use by a different process. You can find the process using that port by running this command and looking for a line with <code>LISTEN</code>:</p>
<pre><code class="lang-auto">netstate -anp | grep 8115
</code></pre>

username: pemfir
cooked: <p>Thank you for your response, i have done this already, changing the port and looking if torch distributed uses 8115, and it does not and it is not seem to be related to the port. I feel it is because FLASK Api launches a server and torch distributed also launch a server and the combination of the two does not work well. Another question is that i want to run inference in a distributed fashion. That is loading several instance of a model and passing a segment of the data to them (propagate or map the models and data), then i have to collect the model predictions from all models (reduce). Does pytorch have any tool or api to support this inference pipeline ?</p>

username: Symbolk
cooked: <p>Just use debug=False instead.</p>

username: MohamedAliRashad
cooked: <p><a class="mention" href="/u/pemfir">@pemfir</a> Did you find a solution to your problem (I have the same issue) ?</p>

username: songruiLI
cooked: <p>The main reason is that when using <code>torch.distributed.lauch</code> to run the model parallel on 2 devices, python generates two processes for each device, and each process runs all the lines in the script.<br>
This would be an issue when it comes to app.run(port=8115), where all the processes would try to take over one same port to launch their own severs.<br>
Imagine process 0 launches <code>app.run</code> on port 8115 first, and successfully.<br>
Process 1 tries to use port 8115 as well, but the port is already taken by process 0.<br>
That’s where the RuntimeError: Address already in use comes from.</p>
<p>I came into this issue as well, I know where it comes from but I don’t know how to solve this problem.</p>

username: wnma3mz
cooked: <p>I have found a less elegant solution that others have implemented. Even though it’s crude. But it works for me.</p>
<aside class="onebox githubpullrequest" data-onebox-src="https://github.com/facebookresearch/llama/pull/147/files">
  <header class="source">

      <a href="https://github.com/facebookresearch/llama/pull/147/files" target="_blank" rel="noopener nofollow ugc">github.com/facebookresearch/llama</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">



    <div class="github-icon-container" title="Pull Request">
      <svg width="60" height="60" class="github-icon" viewbox="0 0 12 16" aria-hidden="true"><path fill-rule="evenodd" d="M11 11.28V5c-.03-.78-.34-1.47-.94-2.06C9.46 2.35 8.78 2.03 8 2H7V0L4 3l3 3V4h1c.27.02.48.11.69.31.21.2.3.42.31.69v6.28A1.993 1.993 0 0 0 10 15a1.993 1.993 0 0 0 1-3.72zm-1 2.92c-.66 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2zM4 3c0-1.11-.89-2-2-2a1.993 1.993 0 0 0-1 3.72v6.56A1.993 1.993 0 0 0 2 15a1.993 1.993 0 0 0 1-3.72V4.72c.59-.34 1-.98 1-1.72zm-.8 10c0 .66-.55 1.2-1.2 1.2-.65 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2zM2 4.2C1.34 4.2.8 3.65.8 3c0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2z"></path></svg>
    </div>

  <div class="github-info-container">



      <h4>
        <a href="https://github.com/facebookresearch/llama/pull/147/files" target="_blank" rel="noopener nofollow ugc">Add simple server</a>
      </h4>

    <div class="branches">
      <code>facebookresearch:main</code> ← <code>zhuzilin:main</code>
    </div>

      <div class="github-info">
        <div class="date">
          opened <span class="discourse-local-date" data-format="ll" data-date="2023-03-07" data-time="09:03:24" data-timezone="UTC">09:03AM - 07 Mar 23 UTC</span>
        </div>

        <div class="user">
          <a href="https://github.com/zhuzilin" target="_blank" rel="noopener nofollow ugc">
            <img alt="zhuzilin" src="https://avatars.githubusercontent.com/u/10428324?v=4" class="onebox-avatar-inline" width="20" height="20">
            zhuzilin
          </a>
        </div>

        <div class="lines" title="1 commits changed 2 files with 143 additions and 1 deletions">
          <a href="https://github.com/facebookresearch/llama/pull/147/files" target="_blank" rel="noopener nofollow ugc">
            <span class="added">+143</span>
            <span class="removed">-1</span>
          </a>
        </div>
      </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">This PR adds a simple fastapi server to serve the llama model.

Thank you for <span class="show-more-container"><a href="https://github.com/facebookresearch/llama/pull/147" target="_blank" rel="noopener nofollow ugc" class="show-more">…</a></span><span class="excerpt hidden">your time on reviewing this PR :)</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


username: yijiong_yu
cooked: <p>I think the best way is wrap the torch process as an individual server, and the flask process sends query to it. Don’t use torch.distributed and flask in the same program.</p>

username: Teja11
cooked: <p>i have been facing this issue while using deepspeed for nvme offloading inference of llm. I am able to get it running for a single gpu serving and facing this issue for multi gpu serving. My concern is to use the deepspeed launcher which in turn uses torch.distributed.launch, for enabling nvme offloading. so, did you get the issue resolved ? if so can you help me ?</p>