Pytorch Model Parallel Best Practices: Pipeline Stats

Vibhatha_Abeykoon: <p>I am trying to replicate the model parallel best practices tutorial.</p>
<p><a href="https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html" rel="noopener nofollow ugc">Model Parallel Pytorch Docs</a></p>
<p>I use Tesla K80 GPUs for running the example. I didn’t plot graphs but I have the following stats.</p>
<p>Single Node Time:  2.1659805027768018<br>
Model Parallel Time:  2.23040875303559<br>
Pipeline 20 Mean: 3.496733816713095</p>
<p>I don’t get the best results at this split size and it could be okay, depending on the hardware, software issues this can be possible. So I went for testing what is going on. Then I ran the rest of the tutorial for different split sizes. These are the results I obtain for the given split sizes.</p>
<p>Here is the graph I obtained,</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/c/7c41530672a7150ea94ae4ea6224c93d7937d2df.png" alt="resnet50_tutorial_v1" data-base62-sha1="hJda6IV0vdUj8gGnxCJYjPixpZZ" width="605" height="340"></p>
<p>Stats for corresponding split sizes</p>
<p>[11.667116858577355, 15.080700974399225, 3.556491438532248, 6.485900523653254, 2.4750063681043684, 4.956193731771782, 3.506740797869861, 1.6466765413060784, 1.5998394633177668]</p>
<p>I don’t get a similar graph. Instead of having a local minima in the middle, I get the minimum value for split size 60.</p>
<p>Then I investigated bit deeper by just logging the times for Forward prop, backward prop, label copy time and optimization time.</p>
<p>I get something like this,</p>
<p>MBi refers to i^th mini-batch<br>
FW : forward time<br>
BW: backward time<br>
LBL_CP: label copy time<br>
OPT : optimization time</p>
<p><strong>For split size 20</strong></p>
<p>MB-1: FW 0.12454676628112793, LBL_CP 0.5665407180786133, BW 0.25083422660827637, OPT 0.015613555908203125<br>
MB-2: FW 0.31687474250793457, LBL_CP 0.5684511661529541, BW 0.26471543312072754, OPT 0.017733335494995117<br>
MB-3: FW 0.3080329895019531, LBL_CP 0.571399450302124, BW 0.2626023292541504, OPT 0.018143177032470703</p>
<p><em><strong>Split Size 1</strong></em></p>
<p>MB1: FW 2.2466013431549072, LBL_CP 0.003688812255859375, BW 1.7002854347229004, OPT 0.0038182735443115234<br>
MB2: FW 2.2562222480773926, LBL_CP 0.00019812583923339844, BW 1.6973598003387451, OPT 0.0039861202239990234<br>
MB3: FW 2.2152814865112305, LBL_CP 0.0023992061614990234, BW 1.6881706714630127, OPT 0.004811525344848633</p>
<p><em><strong>Split Size 3</strong></em></p>
<p>MB1: FW 3.195209264755249, LBL_CP 0.7909142971038818, BW 0.9772884845733643, OPT 0.0038728713989257812<br>
MB1: FW 3.122593402862549, LBL_CP 0.7815954685211182, BW 0.960608959197998, OPT 0.0037987232208251953<br>
MB1: FW 3.2085180282592773, LBL_CP 0.7906265258789062, BW 0.9696476459503174, OPT 0.003855466842651367</p>
<p><em><strong>Split Size 5</strong></em></p>
<p>MB1: FW 0.5092735290527344, LBL_CP 0.003528594970703125, BW 0.6527245044708252, OPT 0.005049228668212891<br>
MB2: FW 0.44788599014282227, LBL_CP 0.0061757564544677734, BW 0.6450486183166504, OPT 0.003782510757446289<br>
MB3: FW 0.514885425567627, LBL_CP 0.003251314163208008, BW 0.6562778949737549, OPT 0.004816293716430664</p>
<p>The label copy time fluctuates a lot, but It always copy the same size of array chunk, isn’ it?</p>
<p>In addition the FW time also fluctuates in an unexpected way. I am trying to profile this and see what happens.</p>
<p>But, I would like to get an insight, why this could be happening?<br>
(Does this has something to do with NVLink?)</p>

aniruddhadave: <p>I am facing a similar issue while trying to replicate the tutorial. In my case the pipelining time is higher than the model-parallel time which is not expected.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/3/b31e5caa52b9c43a08ae2b05b14882d3730b8c4c.png" alt="mp_vs_rn_vs_pp" data-base62-sha1="pyyw52k0hdIuvqoAyXWdyCmoL2Q" width="640" height="480"></p>

Vibhatha_Abeykoon: <p>I also got the same problem. I got a similar graph like this. So I further continued.<br>
It is not clear why it is happening?</p>
<p>I am currently micro-benchmarking this. I still have no clear idea.</p>

mrshenli: <p>Hey <a class="mention" href="/u/vibhatha_abeykoon">@Vibhatha_Abeykoon</a>, how did you log the time for fw, bw and opt? Do you use CUDA events and then call <code>elapsed_time</code>?</p>