Using 2 GPUs for Different Parts of the Model

username: uertenli
cooked: <p>Hi!</p>
<p>I have 2 of the same GPU and I want to achieve faster processing by utilizing both of them. However, I am doing this in a different way, imitating the idea of <a href="https://arxiv.org/pdf/1806.03863.pdf" rel="noopener nofollow ugc">Massively Parallel Video Networks</a>:</p>
<p>I have divided my model into two sub-models. I want to run them concurrently, one part processing the input video frame by frame, and the other processing the output of the first one. However, there is a catch. When the first sub-model returns an output, it passes it to the second sub-model and starts processing the next frame of the input. By utilizing both the GPUs the authors of the paper achieve faster processing. Any idea on how to do this? The figure shows the idea: (the network is unrolled over time)</p>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/c/2c65416531119c70ae45c2590b22acd76711ca03.png" alt="Screenshot%20from%202019-12-20%2021-37-25" data-base62-sha1="6kJWKc98HpySaVORzLwtuVgszPt" width="217" height="489"></p>
<p>The idea is not the same as <code>nn.DataParallel()</code>. I have tried <code>torch.multiprocessing</code>, <code>DistributedDataParallel()</code> but I am having trouble understanding how to do this.</p>
<p>If anyone have some answer, I would be glad.</p>
<p>Thanks.</p>

username: DuaneNielsen
cooked: <p>One approach…</p>
<p>Start 2 python programs, in separate interpreters to avoid the dreaded GIL lock.</p>
<p>Processor 1</p>
<ol>
<li>Put tensor on cuda:0, get the output.</li>
<li>Serialize and push the output to shared redis database</li>
</ol>
<p>Processor 2</p>
<ol>
<li>Consumer picks up from database, pushes to cuda:1</li>
<li>Consumer runs the next step of calculation.</li>
</ol>
<p>If you need to send gradients for backprop you can store and reload them also.</p>
<aside class="quote quote-modified" data-post="1" data-topic="34398">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/duanenielsen/48/4158_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/bit-of-fun-with-gradients/34398">Bit of fun with gradients</a> <a class="badge-category__wrapper " href="/c/autograd/7"><span data-category-id="7" data-drop-close="true" class="badge-category " title="A category of posts relating to the autograd engine itself."><span class="badge-category__name">autograd</span></span></a>
  </div>
  <blockquote>
    Thought I’d share this code.  Learned a few things about autograd doing this. 
import torch

# unbroken gradient, backward goes all the way to x
x = torch.ones(2, 2, requires_grad=True)
y = 2 * x + 2
z = y * y * 3
out = z.mean()
out.backward()
print(x.grad)
baseline_x = x.grad

# broken gradient, ends at _y
x = torch.ones(2, 2, requires_grad=True)
y = 2 * x + 2

_y = torch.tensor(y.detach(), requires_grad=True)
z = _y * _y * 3
out = z.mean()
out.backward()
print(x.grad)
print(_y.grad)

# we can …
  </blockquote>
</aside>

<p>That’s one way…  Not easy though.  I spent easy a month just trying to distribute calculations over multiple processors.</p>
<p>If you can pull it off… then it’s an awesome skill.</p>
<p>Also, there is the Ray project <a href="https://github.com/ray-project/ray" class="inline-onebox" rel="noopener nofollow ugc">GitHub - ray-project/ray: Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.</a></p>
<p>I tried using it.  It had great promise, but ended up being a bit too new at the time.  It might be a bit more mature now.</p>

username: uertenli
cooked: <p>Thanks for your reply and sorry for my late reply. I will look into these methods. I am only doing this for the test phase, so I will only have to transfer one tensor per input frame to processor 2.</p>
<p>If anybody else has some further suggestions, I will be happy to hear them!</p>
<p>Thanks.</p>

username: mrshenli
cooked: <p>Will this tutorial be helpful? <a href="https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html" rel="nofollow noopener">https://pytorch.org/tutorials/intermediate/model_parallel_tutorial.html</a></p>

username: uertenli
cooked: <p>Thank you. I have seen this tutorial previously, however, the model parallel part is not what I want. For the pipelining part, I am having trouble how that part is getting executed. If you can further clarify that part for me, that would be great.</p>

username: MeinhardMark
cooked: <p>Did you find a solution?<br>
I have the same problem and need to seperate my model into parts and load them on 2 GPUs. But the second part is a frozen LLM and it doesn’t require grad.</p>