How do I run Inference in parallel?

username: Vivek_Reddy
cooked: <p>Hello,<br>
I have 4 GPUs available to me, and I’m trying to run inference utilizing all of them.  I’m confused by so many of the multiprocessing methods out there (e.g. Multiprocessing.pool, torch.multiprocessing, multiprocessing.spawn, launch utility).<br>
I have a model that I trained.  However, I have several hundred thousand crops I need to run on the model so it is only practical if I run processes simultaneously on each GPU.  I have 4 GPUs available to me.  I would like to assign one model to each GPU and run 1/4 the data on each.  How can I do this?<br>
Thank you in advance.</p>

username: wayi
cooked: <p>Since parallel inference does not need any communication among different processes, I think you can use any utility you mentioned to launch multi-processing. We can decompose your problem into two subproblems: 1) launching multiple processes to utilize all the 4 GPUs; 2) Partition the input data using <code>DataLoader</code>.</p>
<pre><code class="lang-auto">import torch
import torch.distributed as dist
import torch.multiprocessing as mp

def run_inference(rank, world_size):
    # create default process group
    dist.init_process_group("gloo", rank=rank, world_size=world_size)
    
    # load a model 
    model = YourModel()
    model.load_state_dict(PATH)
    model.eval()
    model.to(rank)

    # create a dataloader
    dataset = ...
    loader = torch.utils.data.DataLoader(dataset=dataset,
                                               batch_size=batch_size,
                                               shuffle=True,
                                               num_workers=4)

    # iterate over the loaded partition and run the model
    for idx, data in enumerate(loader):
            ...

def main():
    world_size = 4
    mp.spawn(run_inference,
        args=(world_size,),
        nprocs=world_size,
        join=True)

if __name__=="__main__":
    main()
</code></pre>

username: Vivek_Reddy
cooked: <p>Thank you.  I will try this out now.  I’m assuming that “example” in mp.spawn is the run_inference function?<br>
Also, is it possible to make each GPU run multiple processes or no?</p>

username: wayi
cooked: <blockquote>
<p>I’m assuming that “example” in mp.spawn is the run_inference function?</p>
</blockquote>
<p>Yes, that’s a typo. Fixed now.</p>
<blockquote>
<p>Also, is it possible to make each GPU run multiple processes or no?</p>
</blockquote>
<p>Running multiple processes on each GPU will be slower, so not recommended IMO.</p>

username: seungjun
cooked: <p>I recommend using a custom sampler.<br>
Related thread: <a href="https://discuss.pytorch.org/t/distributedsampler/90205/3">DistributedSampler</a></p>
<p>By default, DistributedSampler divides the dataset by the number of processes (equivalent to <span class="hashtag">#GPUs</span>).<br>
In the above thread, I provided an example modification on the sampler to avoid duplication of data.</p>

username: Vivek_Reddy
cooked: <p>Would the above code stay the same, and I would add the DistributedSampler to verify that each process is getting an equal split of different data?</p>

username: seungjun
cooked: <p>DistributedSampler with modification will give you the almost equal-sized splits.</p>
<ul>
<li>I don’t know how you defined your model but you should also use DDP to maximally parallelize the models with multiple GPUs &amp; use DistributedSampler with multiple processes.</li>
<li>make sure to customize the sampler so that there is no overlap between the different ranks (processes).</li>
<li>you should communicate between different processes to collect loss or accuracy metrics.</li>
</ul>
<p>You may want to take a look at my <a href="https://github.com/SeungjunNah/DeepDeblur-PyTorch" rel="noopener nofollow ugc">github repository</a> for an example.</p>

username: wayi
cooked: <blockquote>
<ul>
<li>I don’t know how you defined your model but you should also use DDP to maximally parallelize the models with multiple GPUs &amp; use DistributedSampler with multiple processes.</li>
</ul>
</blockquote>
<p>Do you mean using DDP for inference for this case?</p>

username: seungjun
cooked: <p><a class="mention" href="/u/wayi">@wayi</a><br>
Fix: multiprocessing without DDP can also work if limited to inference only.</p>
<p>It is my preference is to use DDP at inference, too, because I don’t want to change my model object at training time which is DDP.</p>

username: Andrei_Pokrovsky
cooked: <p>There’s no communication between processes during inference, I don’t think you need gloo here. You can just run n processes with different CUDA_VISIBLE_DEVICES.</p>

username: zhangdan8962
cooked: <p>I am wondering what would be the difference if I add the DistributedSampler to the dataloader here?</p>