Evaluate multiple models on multiple GPUs

username: Zdeeno
cooked: <p>Hello guys,</p>
<p>I would like to do parallel evaluation of my models on multiple GPUs.  I don’t have much experience using python and pytorch this way. Here is a pseudocode of what I’m trying to do:</p>
<pre><code class="lang-auto">import torch
import torch.multiprocessing as mp
from mycnn import CNN
from data_parser import parser
from fitness import get_fitness  # this also runs on GPU

def run_model(outputs, model, device_id, input):
    out = model(input)
    f = get_fitness(out)    # due to this I cannot just run: model(input, non_blocking=True)
    outputs[device_id] = f.cpu()

if __name__ == '__main__':

batch = parser.get_batch()
model = CNN()
GPU_NUM = 2
outputs = torch.zeros(GPU_NUM, dtype=torch.double)  # collect outputs here
outputs.share_memory_()     # I guess this is not enough to make it work
mp.set_start_method('spawn')
processes = []

for dev_id in (range(GPU_NUM)):
    device = torch.device("cuda:" + str(dev_id))
    dev_model = model.to(device)
    dev_batch = batch.to(device)
    p = mp.Process(target=run_model, args=(outputs, dev_model, dev_id, dev_batch))
    p.start()
    processes.append(p)
for p in processes:
    p.join()

print(outputs)
</code></pre>
<p>Sadly this doesn’t work at all and I’m probably doing it completely wrong. I’m getting this error:</p>
<pre><code class="lang-auto">OSError: [Errno 12] Cannot allocate memory
</code></pre>
<p>For some reason it drains all memory on my server, even when GPU_NUM = 1. When I run code synchronously I get no errors. Could you please tell me what is the right way to do something like this or give me link to some examples, which would help me?</p>

username: ptrblck
cooked: <p>If you have multiple GPUs and want to evaluate each model on a single dedicated GPU independently, you could just push each model to a GPU via:</p>
<pre><code class="lang-python">modelA = modelA.to('cuda:0')
modelB = modelB.to('cuda:1')
...
</code></pre>
<p>and evaluate each model separately.<br>
Since CUDA calls are asynchronous, the GPUs shouldn’t block each other, if you don’t synchronize them manually.</p>

username: rvarm1
cooked: <aside class="quote no-group" data-username="Zdeeno" data-post="1" data-topic="62451">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/zdeeno/48/18243_2.png" class="avatar"> Zdeeno:</div>
<blockquote>
<pre><code class="lang-auto">OSError: [Errno 12] Cannot allocate memory
</code></pre>
</blockquote>
</aside>
<p>If the cannot allocate memory issues still persist, you might try allocating additional swap space on your machine (see <a href="https://github.com/pytorch/pytorch/issues/4387" class="inline-onebox" rel="noopener nofollow ugc">cannot allocate memory · Issue #4387 · pytorch/pytorch · GitHub</a> for more details).</p>

username: r2d2bol
cooked: <p>what if you have a large amount of data to process (evaluate, not train) and you want to take advantage of mulitple gpus?  in this case would it be best to use the DataParallel class and just use a larger batch size?  or would replicating the model using something similar to the example:</p>
<p>modelA = modelA.to(‘cuda:0’)<br>
modelB = modelB.to(‘cuda:1’)</p>
<p>be the way to go?</p>

username: ptrblck
cooked: <p>If the model fits on a single GPU, you could use a data parallel approach such as <code>nn.DataParallel</code>.<br>
Model sharding could also be used to increase the batch size, but I think a data parallel approach might be easier to apply, since splitting the model is two parts which have approx. an equal memory usage might be more tricky.</p>
<p>Note that if you are using<code>DistributedDataParallel</code> during the evaluation, you would have to make sure the complete validation set is used only once. By default the <code>DistributedSampler</code> will add extra samples to make the dataset evenly divisible as seen in <a href="https://github.com/pytorch/pytorch/blob/13b7855f33e61e3c466197518250e57ffe5886db/torch/utils/data/distributed.py#L102">this line of code</a>.</p>