Most effective multiple model inference

sebastienwood: <p>Hi all ! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=7" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>
<p>I’m running across an issue for a test scenario I need to run. I have a dozen of models built with the exact same pattern (WideResNet 28-10). Each of those model have slight differences due to training divergences.<br>
I however would need to qualify each of those model on the entire test set (4 minibatches, CIFAR 10). Those models may eventually get back into a training iteration following the test.</p>
<p>The easy way to go would be to do a repeated iteration in the test set on each of those models, i.e. roughly :</p>
<pre><code class="lang-auto">for i, (input, target) in enumerate(test_loader):
      target, input = target.to(device), input.to(device)
      for model in models:
           output = model(input)
           loss = criterion(output, target)
           acc1 = accuracy(output, target) 
           # metric management
</code></pre>
<p>Which seems a bit inefficient/slow (2 min per minibatch, 4 minibatches, a thousand models … <img src="https://discuss.pytorch.org/images/emoji/apple/sob.png?v=7" title=":sob:" class="emoji" alt=":sob:"> ). Is there any way to do so in a clean way which would exploit the embarassingly parallel nature of this task ?</p>
<p>At first I thought to concat the models (in the end its just sets of layers that could operate in parallel ?) together but I have no clue on how to achieve that in Pytorch/don’t even know if its doable. Multiprocessing also appeared as a possible choice but would it be any efficient ? Any suggestion is welcome <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=7" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

Kushaj: <pre><code class="lang-python">class MyModel(nn.Module):
    def __init__(self, model1, model2, model3):
        super().__init__()
        # You can also pass a list, rather than 
        # seperate models
        self.model1 = model1
        self.model2 = model2
        self.model3 = model3
    def forward(self, x):
        out1 = self.model1(x)
        out2 = self.model2(x)
        out3 = self.model3(x)
        return out1, out2, out3
</code></pre>
<p>This method involves moving all the models to GPU. So GPU memory would become bottleneck, but you will only have one set of inputs.</p>
<p>You can also do inference on CPU, which may be faster if your GPU RAM is small.</p>

sebastienwood: <p>Thanks for your suggestion. Do you know if Pytorch would run some optimization in the background thus leading to parallelization in your suggestion ?<br>
By reading it/not being perfect on Pytorch background optims I fear that it would eventually run the models in a sequence (like a for loop would) rather than in parallel.</p>
<p>The goal is to have time_parallel(model 1, …, model k) &lt;&lt; time_unary(model 1) + … + time_unary(model k).</p>

Kushaj: <p>For GPU, forget about multiprocessing. It is a very tedious task.<br>
For CPU you can use torch.multiprocessing.</p>

guillaume_bertand: <p>Hello Kushaj,</p>
<p>the problem is that this solution runs the 3 models sequentially, not in parallel</p>