Model Parallel Pipelining not working

aniruddhadave: <p>I tried to reproduce the results by using the code provided in the tutorial on <a href="https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html" rel="noopener nofollow ugc">Single Machine Model Parallel Best Practices</a>, however the results were a bit different.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b31e5caa52b9c43a08ae2b05b14882d3730b8c4c.png" alt="mp_vs_rn_vs_pp" data-base62-sha1="pyyw52k0hdIuvqoAyXWdyCmoL2Q" width="640" height="480"><br>
The model with pipelining is expected to perform better than than the rest two cases but it doesn’t. What can be the possible reasons for this?</p>

mrshenli: <p>Hey <a class="mention" href="/u/aniruddhadave">@aniruddhadave</a>, a lot of configurations could affect the performance, e.g., split size, hardware type, GPU interconnection bandwidth, model complexity, etc. And it does takes effort to get the best performance. One place to start with could be drawing the split size curve using your environment. I mean this figure blow. How does it look on your side?</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/7/176aad2cccad5ce4d015797d0729dc44980d92f2.png" alt="image" data-base62-sha1="3l9xaTrBu887CYdnc45yQsVmLse" width="640" height="480"></p>

Vibhatha_Abeykoon: <p><a class="mention" href="/u/mrshenli">@mrshenli</a></p>
<p>I also observed a similar result.</p>
<p>This is the thread I created in the discussion.</p>
<aside class="quote quote-modified" data-post="1" data-topic="69380">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vibhatha_abeykoon/48/17261_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/pytorch-model-parallel-best-practices-pipeline-stats/69380">Pytorch Model Parallel Best Practices: Pipeline Stats</a> <a class="badge-category__wrapper " href="/c/distributed/12"><span data-category-id="12" data-drop-close="true" class="badge-category "><span class="badge-category__name">distributed</span></span></a>
  </div>
  <blockquote>
    I am trying to replicate the model parallel best practices tutorial. 
<a href="https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html" rel="noopener nofollow ugc">Model Parallel Pytorch Docs</a> 
I use Tesla K80 GPUs for running the example. I didn’t plot graphs but I have the following stats. 
Single Node Time:  2.1659805027768018 
Model Parallel Time:  2.23040875303559 
Pipeline 20 Mean: 3.496733816713095 
I don’t get the best results at this split size and it could be okay, depending on the hardware, software issues this can be possible. So I went for testing what is going on. Then I ran…
  </blockquote>
</aside>

<p>I also benchmarked this using multiple configurations.</p>
<p>I am not sure about the concurrent run of the code. So I changed it as follows and got a sort of fine result.</p>
<pre data-code-wrap="python"><code class="lang-python">class PipelineParallelResNet50(ModelParallelResNet50):
    def __init__(self, split_size=20, *args, **kwargs):
        super(PipelineParallelResNet50, self).__init__(*args, **kwargs)
        self.split_size = split_size

    def taskA(self, s_prev, ret):
        s_prev = self.seq2(s_prev)
        ret.append(self.fc(s_prev.view(s_prev.size(0), -1)))

    def taskB(self, s_next):
        s_prev = self.seq1(s_next).to('cuda:1')
        return s_prev

    def forward(self, x):
        splits = iter(x.split(self.split_size, dim=0))
        s_next = next(splits)
        s_prev = self.seq1(s_next).to('cuda:1')
        ret = []

        for s_next in splits:
            # A. s_prev runs on cuda:1
            # self.taskA(s_prev=s_prev, ret=ret)
            with concurrent.futures.ThreadPoolExecutor() as executor:
                futureA = executor.submit(self.taskA, s_prev, ret)
                futureA.result()

            # B. s_next runs on cuda:0, which can run concurrently with A
            with concurrent.futures.ThreadPoolExecutor() as executor:
                futureB = executor.submit(self.taskB, s_next)
                s_prev = futureB.result()

        s_prev = self.seq2(s_prev)
        ret.append(self.fc(s_prev.view(s_prev.size(0), -1)))

        return torch.cat(ret)
</code></pre>

mrshenli: <p>Hey <a class="mention" href="/u/vibhatha_abeykoon">@Vibhatha_Abeykoon</a></p>
<p>Based on the numbers you posted in that thread, looks like you get the best performance with <code>split_size=60</code>, which gives you around 1.8s execution time, and it is a little faster than the 2.2 single node time?</p>
<p>BTW are you using the same model as the tutorial?</p>

aniruddhadave: <p>Hey, Thank you for the response. The split size was indeed the reason for the difference inperformance. On changing the split size I was able to reduce the time for pipelined model.</p>

Vibhatha_Abeykoon: <p><a class="mention" href="/u/mrshenli">@mrshenli</a></p>
<p>I am exactly using the same code. I couldn’t replicate similar results.<br>
Correct me if I have misunderstood the concept of running two micro-batches concurrently,</p>
<pre><code class="lang-python"> for s_next in splits:
            # A. s_prev runs on cuda:1
            s_prev = self.seq2(s_prev)
            ret.append(self.fc(s_prev.view(s_prev.size(0), -1)))

            # B. s_next runs on cuda:0, which can run concurrently with A
            s_prev = self.seq1(s_next).to('cuda:1') 
</code></pre>
<p>Here part A, and part B as shown in the comments must run concurrently to get the pipeline performance. Am I correct/wrong?</p>
<p>If so, as Pytorch eagerly executes the layers, it is not asynchronous? Am I following this right?</p>
<p>If both these clauses are true, having threads is required? Isn’t it?</p>
<p>My reasoning comes with the usage of the for loop. Here within the loop a concurrent execution could<br>
happen if we use threads. Am I following this wrong?</p>

mrshenli: <aside class="quote no-group" data-username="Vibhatha_Abeykoon" data-post="6" data-topic="69322">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vibhatha_abeykoon/48/17261_2.png" class="avatar"> Vibhatha_Abeykoon:</div>
<blockquote>
<p>Here part A, and part B as shown in the comments must run concurrently to get the pipeline performance. Am I correct/wrong?</p>
</blockquote>
</aside>
<p>No. Because CUDA operations run asynchronously from CPU’s point of view, unless you explicitly call <code>synchronize()</code> on CPU. And they are inserted into the same CUDA stream on each device in this example, which will guarantee ops on the same device run in order, but ops on different device can run in parallel.</p>
<aside class="quote no-group" data-username="Vibhatha_Abeykoon" data-post="6" data-topic="69322">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vibhatha_abeykoon/48/17261_2.png" class="avatar"> Vibhatha_Abeykoon:</div>
<blockquote>
<p>If so, as Pytorch eagerly executes the layers, it is not asynchronous? Am I following this right?</p>
</blockquote>
</aside>
<p>Yes, it launches the CUDA kernel right away, but the CUDA kernel execution can be asynchronous. And CUDA will actually queue the kernels in the stream and to coordinate the execution. So, “launch right away” does not mean it will wait for the kernel to finish, nor does it mean the the kernel will start on GPU immediately.</p>
<aside class="quote no-group" data-username="Vibhatha_Abeykoon" data-post="6" data-topic="69322">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/vibhatha_abeykoon/48/17261_2.png" class="avatar"> Vibhatha_Abeykoon:</div>
<blockquote>
<p>If both these clauses are true, having threads is required? Isn’t it?</p>
</blockquote>
</aside>
<p>No, with multiple CUDA device + async CUDA kernel behavior, you could still get parallel execution without threads. And actually, even with thread, all Python code on CPU will still run in sequentially due to GIL.</p>

Vibhatha_Abeykoon: <p><a class="mention" href="/u/mrshenli">@mrshenli</a></p>
<p>Yes, there is a part that GIL avoids. I was trying to use multiprocessing (torch version), it gave some memory issues. I understand your point.</p>
<p>I was trying to run this on K80 GPUs. I didn’t get the graph as shown in the tutorial. Does old hardware could be an issue for not getting the expected graph?</p>
<p>Could you share the script run command, I assumed it is just python scripy.py<br>
do I have to use any specific flags or CUDA env variables that need to be set.</p>
<p>I am just curious why the performance is not there as expected in the tutorial.</p>

mrshenli: <blockquote>
<p>Could you share the script run command, I assumed it is just python scripy.py<br>
do I have to use any specific flags or CUDA env variables that need to be set.</p>
</blockquote>
<p>I was actually using the exactly same script as shown in that tutorial. It is a <code>.rst</code> instead of a notebook because I don’t know what the hardware spec of our tutorial servers will be. I would like to avoid tuning parameters every time the underlying server hardware changes. I probably should have highlighted in the very beginning saying that the result will be different in different envs, and each env would require efforts to explore configuration space to get the best perf. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<p>I did that a while back, and my server env should be either this:</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------+
| NVIDIA-SMI 396.69                 Driver Version: 396.69                    |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Tesla M40           On   | 00000000:0B:00.0 Off |                    0 |
|  0%   27C    P8    18W / 250W |      0MiB / 11448MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Tesla M40           On   | 00000000:0D:00.0 Off |                    0 |
|  0%   25C    P8    18W / 250W |      0MiB / 11448MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
</code></pre>
<pre><code class="lang-auto">        GPU0    GPU1    CPU Affinity
GPU0     X      PIX     0-11,24-35
GPU1    PIX      X      0-11,24-35
</code></pre>
<p>or this:</p>
<pre><code class="lang-auto">+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.116.00   Driver Version: 418.116.00   CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro GP100        On   | 00000000:81:00.0 Off |                    0 |
| 26%   32C    P0    29W / 235W |      1MiB / 16278MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  Quadro GP100        On   | 00000000:82:00.0 Off |                    0 |
| 26%   33C    P0    30W / 235W |      1MiB / 16278MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+
</code></pre>
<pre><code class="lang-auto">	GPU0	GPU1	CPU Affinity
GPU0	 X 	NV4	12-23
GPU1	NV4	 X 	12-23
</code></pre>

Vibhatha_Abeykoon: <p><a class="mention" href="/u/mrshenli">@mrshenli</a></p>
<p>Thanks a lot for finding the machine configs. I will see if I can run in modern hardware.</p>
<p>For K80s that was the best I was able to produce. I did some micro-benchmarks as well to understand some of the bottlenecks. But for my experiments I got the optimum results close to end of the curve (end of the curve meaning the graphs +infinity direction).</p>
<p>But I didn’t get a significant speed up from the code. That’s why I was trying different approaches.</p>