Split Single GPU

ahkarami: <p>Is there any way to <em>split single GPU</em> and use a single GPU as multiple GPUs?<br>
For example, we have 2 different <em>ResNet18</em> model and we want to forward pass these two models in parallel just in one GPU (with enough memory, e.g., 12Gb). I mean that the <em>forward pass</em> of these two models runs in parallel and concurrent in just one GPU.</p>

vince62s: <p>if your code is Torch Distributed compatible, you can spawn two processes on the same device.</p>

Naman-ntc: <p>I am not sure if I understood your question properly.<br>
Can’t you just do<br>
<code>python model1.py</code><br>
<code>python model2.py</code><br>
in two different shells??</p>
<p>(Although this slows down both individual processes, havent seen any recommended method that maintains the speed!)</p>

roaffix: <p>I’m not sure if I get what you want, but possibly my last topic (<a href="https://discuss.pytorch.org/t/couple-of-models-in-production/18501?u=roaffix">Couple of models in production</a>) will guide you.</p>
<p>Cheers,<br>
Anton</p>

justusschock: <p>You could try using data parallel (as for multiple GPUs) and pass the same GPU ID several times</p>

ahkarami: <p>Thanks all for above replies.<br>
I have read the topic (<a href="https://discuss.pytorch.org/t/couple-of-models-in-production/18501">Couple of models in production</a>) and according it I have implemented these codes:<br>
<strong>First Scenario (Sequential Forward Pass):</strong></p>
<pre><code class="lang-python">import torch
import time
from torchvision import models
from torch.autograd import Variable

# Check use GPU or not
use_gpu = torch.cuda.is_available()  # use GPU

torch.manual_seed(123)
if use_gpu:
    torch.cuda.manual_seed(456)


# Define CNN Models:
model1 = models.resnet18(pretrained=True)
model2 = models.resnet50(pretrained=True)

# Eval Mode:
model1.eval()
model2.eval()

# Put on GPU:
if use_gpu:
    model1 = model1.cuda()
    model2 = model2.cuda()

# Create tmp Variable:
x = Variable(torch.randn(10, 3, 224, 224))
if use_gpu:
    x = x.cuda()


# Forward Pass:
tic1 = time.time()
out1 = model1(x)
out2 = model2(x)
tic2 = time.time()

sequential_forward_pass = tic2 - tic1
print('Time = ', sequential_forward_pass)  # example output --&gt; Time =  0.6485
</code></pre>
<p>Now I want to perform the forward passes in parallel in just one single GPU.<br>
<strong>Second Scenario (Parallel Forward Pass):</strong></p>
<pre><code class="lang-python">import time
import torch
from torchvision import models
import torch.multiprocessing as mp
from torch.autograd import Variable

# Check use GPU or not
use_gpu = torch.cuda.is_available()  # use GPU

torch.manual_seed(123)
if use_gpu:
    torch.cuda.manual_seed(456)


# Define Forward Pass Method:
def forward_pass_method(model, tmp_variable):
    output = model(tmp_variable)
    return output


# Define CNN Models:
model1 = models.resnet18(pretrained=True)
model2 = models.resnet50(pretrained=True)

# Eval Mode:
model1.eval()
model2.eval()

# Put on GPU:
if use_gpu:
    model1 = model1.cuda()
    model2 = model2.cuda()

# Create tmp Variable:
x = Variable(torch.randn(10, 3, 224, 224))
if use_gpu:
    x = x.cuda()

# Parallelized the Forward Passes:
tic1 = time.time()

model1.share_memory()
model2.share_memory()

processes = []
num_processes = 2
for i in range(num_processes):
    if i == 0:
        p = mp.Process(target=forward_pass_method, args=(model1, x))
    else:
        p = mp.Process(target=forward_pass_method, args=(model2, x))
    p.start()
    processes.append(p)
for p in processes:
    p.join()

tic2 = time.time()

parallel_forward_pass = tic2 - tic1
print('Time = ', parallel_forward_pass)
</code></pre>
<p>However the second method has the below error:<br>
<code>...RuntimeError: CUDA error (3): initialization error</code><br>
Would you please kindly help me to address the error?<br>
However, It is worth nothing that, I am in doubt that parallelizing by just one single GPU is a feasible task or not.</p>

ahkarami: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
Do you have any idea about my last post?<br>
I have followed <a href="https://discuss.pytorch.org/t/couple-of-models-in-production/18501/2">your opinion</a>.</p>

ptrblck: <p>The error seems to be related to some issues with multiprocessing and CUDA.<br>
Have a look at the doc on <a href="https://pytorch.org/docs/master/notes/multiprocessing.html#sharing-cuda-tensors">Sharing CUDA tensors</a>.</p>
<p>You have to use the “spawn” or “forkserver” start method.</p>
<p>Also, in your first script your time measurement is a bit wrong, because you have to call <code>torch.cuda.synchronize()</code> before getting the end time.<br>
CUDA calls are asynchronous, so that the end time might be stored before the CUDA operation is done.</p>
<p>Here is a small script for your second use case, which might be a starter (I’m not sure, if you need <code>modelX.share_memory()</code>):</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
import torch.optim as optim

import torch.multiprocessing as _mp
import torchvision.datasets as datasets
import torchvision.transforms as transforms
from torch.utils.data import DataLoader


# Globals
mp = _mp.get_context('spawn')
use_cuda = True


class Flatten(nn.Module):
    def __init__(self):
        super(Flatten, self).__init__()

    def forward(self, x):
        x = x.view(x.size(0), -1)
        return x


def get_model():
    model = nn.Sequential(
            nn.Conv2d(3, 6, 3 ,1, 1),
            nn.ReLU(),
            nn.MaxPool2d(2),
            nn.Conv2d(6, 16, 3, 1, 1),
            nn.ReLU(),
            nn.MaxPool2d(2),
            nn.Conv2d(16, 1, 3, 1, 1),
            nn.MaxPool2d(2),
            Flatten(),
            nn.Linear(28*28, 10),
            nn.LogSoftmax(dim=1)
    )
    
    return model


def train(model, data_loader, optimizer, criterion):
    for data, labels in data_loader:
        labels = labels.long()
        if use_cuda:
            data, labels = data.to('cuda'), labels.to('cuda')
        optimizer.zero_grad()
        output = model(data)
        loss = criterion(output, labels)
        loss.backward()
        optimizer.step()


if __name__=='__main__':
    num_processes = 2
    model1 = get_model()
    model2 = get_model()
    if use_cuda:
        model1 = model1.to('cuda')
        model2 = model2.to('cuda')
    
    dataset = datasets.FakeData(transform=transforms.ToTensor())
    data_loader = DataLoader(dataset, batch_size=2,
                             num_workers=0,
                             pin_memory=False)
    
    criterion = nn.NLLLoss()
    optimizer1 = optim.SGD(model1.parameters(), lr=1e-3)
    optimizer2 = optim.SGD(model2.parameters(), lr=1e-3)
    
    #model1.share_memory()
    #model2.share_memory()
    processes = []
    p1 = mp.Process(target=train, args=(model1, data_loader, optimizer1, criterion))
    p1.start()
    processes.append(p1)
    p2 = mp.Process(target=train, args=(model2, data_loader, optimizer2, criterion))
    p2.start()
    processes.append(p2)

    for p in processes:
        p.join()
    
    print('Done')
</code></pre>
<p>However, I’m still not sure, if you’ll see any performance advantage.<br>
It would be nice, if you could time your script and report the results using the sequential and multiprocessing way.</p>

ahkarami: <p>Dear <a class="mention" href="/u/ptrblck">@ptrblck</a>,<br>
Thank you for your time &amp; response. I have used the <strong>spawn</strong> start method, and the errors have addressed. Now, my modified code is as below:</p>
<pre><code class="lang-python">import time
import torch
from torchvision import models
import torch.multiprocessing as mp
from torch.autograd import Variable

# Check use GPU or not
use_gpu = torch.cuda.is_available()  # use GPU

torch.manual_seed(123)
if use_gpu:
    torch.cuda.manual_seed(456)

# spawn start method:
mp = mp.get_context('spawn')


# Define Forward Pass Method:
def forward_pass_method(model, tmp_variable):
    output = model(tmp_variable)
    return output


# Define CNN Models:
model1 = models.resnet18(pretrained=True)
model2 = models.resnet50(pretrained=True)

# Eval Mode:
model1.eval()
model2.eval()

# Put on GPU:
if use_gpu:
    model1 = model1.cuda()
    model2 = model2.cuda()

# Create tmp Variable:
x = Variable(torch.randn(10, 3, 224, 224))
if use_gpu:
    x = x.cuda()

# model1.share_memory()
# model2.share_memory()

if __name__ == '__main__':

    # Parallelized the Forward Passes:
    tic1 = time.time()
    processes = []
    num_processes = 2
    for i in range(num_processes):
        if i == 0:
            p = mp.Process(target=forward_pass_method, args=(model1, x))
        else:
            p = mp.Process(target=forward_pass_method, args=(model2, x))
        p.start()
        processes.append(p)
    for p in processes:
        p.join()

    tic2 = time.time()

    parallel_forward_pass = tic2 - tic1
    print('Time = ', parallel_forward_pass)

</code></pre>
<p>However the computational time increased a lot (both qualitatively and quantitatively) in comparison to the <em>sequential way</em>. As a result, I came to the conclusion that the <em>multiprocessing usage of GPU</em> hasn’t any performance advantage.</p>

ptrblck: <p>Have you tried in with a larger workload or just a single forward pass?<br>
I think the startup might take much longer in the multi-processing case, so it’s maybe still faster in the long run.<br>
But as I said, it’s a lot of speculation, since I haven’t used this approach yet.</p>

elepherai: <p><a class="mention" href="/u/ahkarami">@ahkarami</a> <a class="mention" href="/u/ptrblck">@ptrblck</a>  Failed to use this code to inference multiple models on single GPU on windows.</p>
<p>RuntimeError: cuda runtime error (71) : operation not supported at C:\w\1\s\windows\pytorch\torch/csrc/generic/StorageSharing.cpp:245</p>
<p>Could you pls help me to solve this problem?</p>

ahkarami: <p>Unfortunately I haven’t worked PyTorch in Windows.</p>

ptrblck: <p>Your error seems to be <a href="https://discuss.pytorch.org/t/runtimeerror-cuda-runtime-error-71-operation-not-supported-at/40180">this one</a>.</p>
<aside class="quote no-group" data-username="peterjc123" data-post="2" data-topic="40180">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/peterjc123/48/2790_2.png" class="avatar"><a href="https://discuss.pytorch.org/t/runtimeerror-cuda-runtime-error-71-operation-not-supported-at/40180/2">RuntimeError: cuda runtime error (71) : operation not supported at </a></div>
<blockquote>
<p><a href="https://pytorch.org/docs/stable/notes/windows.html#multiprocessing-error-without-if-clause-protection">https://pytorch.org/docs/stable/notes/windows.html#multiprocessing-error-without-if-clause-protection </a></p>
</blockquote>
</aside>

peterjc123: <p>Actually, you can’t share models across processes on Windows, but you can share tensors instead.</p>

elepherai: <p>Thanks ptrblck.</p>
<p>I have used if-clause protection, but still run into this issue.</p>

elepherai: <p>Thanks peterjc123,</p>
<p>Could you share more details?</p>
<p>I’m using three instance segmentation models which were trained by maskrcnn-benchmark, and would like to inference by using multiprocessing.</p>
<p>How to share tensors across processes on Windows?</p>

peterjc123: <p>What about starting three web servers with different ports?</p>

elepherai: <p>That is a way, but I just want to run locally.</p>

peterjc123: <p>That is also possible, use multiprocessing and pass model paths and two queues (input/output) as arguments and then push tensors into the input queue and retrieve the answers from the output queue.</p>

elepherai: <p>Thanks!</p>
<p>What’s diffenrence between this solution and multiprocessing.Pool?</p>
<p>Do you mean that I should load a single model in each process and use the producer-consumer pattern like queues to input/output?</p>