Load multiple models for multi process inference

Bruce_Dong: <p>here is my code ,it worked in parallel, but each inference  took a hundredfold longer.</p>
<p>without multi process ,each  inference cost  20ms,<br>
with multi process,  each inference cost 25s.</p>
<pre><code class="lang-auto">def load_model():
    conf_obj = _get_config_obj("tests/ocr/ocr.yml")
    model_manange = OCRModel(conf_obj)
    model_manange.load_model()
    model_manange.status = MODEL_MANAGE_IDLE
    return model_manange


def init_multi_load() -&gt; None:
    for i in range(3):
        model_manage_list.append(load_model())


def infer(data: dict, model_manage_list) -&gt; dict:
    for model_manage in model_manage_list:
        if model_manage.status == MODEL_MANAGE_IDLE:
            model_manage.status = MODEL_MANAGE_WORKING
            ret = model_manage.run_model(data)
            model_manage.status = MODEL_MANAGE_IDLE
            return ret
    raise ValueError("no idle model for service")


if __name__ == '__main__':
    torch.multiprocessing.set_start_method('spawn', force=True)
    model_manage_list = Manager().list()
    init_multi_load()
    data = {} # just for show case here
    for i in range(3):
        mp = Process(target=infer, args=(data, model_manage_list,))
        mp.start()

    while True:
        time.sleep(2)
</code></pre>

eqy: <p>What hardware is running the model? If the processes are e.g., sharing a single GPU the increased contention could slow things down</p>

Bruce_Dong: <p><img src="https://discuss.pytorch.org/images/emoji/apple/pray.png?v=12" title=":pray:" class="emoji" alt=":pray:" loading="lazy" width="20" height="20"> i got just single GPU in my computer<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2f0812c55783e0de1f73a6ee226b1ed2b50d0de.png" data-download-href="https://discuss.pytorch.org/uploads/default/d2f0812c55783e0de1f73a6ee226b1ed2b50d0de" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/2/d2f0812c55783e0de1f73a6ee226b1ed2b50d0de.png" alt="image" data-base62-sha1="u63xWxkrVb39V7QY8lXoHyp7MWW" width="690" height="262" data-dominant-color="F3F3F3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">727×277 21.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Is there a way to parallelize on single GPU ?    I think my GPU performance is very redundant</p>

eqy: <p>How similar are the different models on your GPU? If they are similar and have relatively simple building blocks, you might look into e.g., using batched matmul in place of linear layers <a href="https://pytorch.org/docs/stable/generated/torch.bmm.html" class="inline-onebox" rel="noopener nofollow ugc">torch.bmm — PyTorch 1.13 documentation</a> and grouped convolutions in place of “vanilla” convolutions to do multiple models worth of computation at the time/per layer. Of course, you would need to be careful about keeping normalization statistics separate across the models.</p>

Bruce_Dong: <p>Hi Dear , Exactly the same model, torch.bmm doesn’t work for me. Do you have such an example ? that  load  multi model in one process and inference with  subprocess parallel.</p>