Several models loaded but just one at a time in GPU

WaterKnight: <p>Hey guys,</p>
<p>I am doing a website for comparing mask outputs of several models. However, this models are very big and just one model fits in my GPU at a time?</p>
<p>I am passing the image throught several models sequently.</p>
<p>Is there a way of moving to GPU a model, make inference and after that move model again to GPU?</p>

bonzogondo: <p>Hi,</p>
<p>What comes to me mind (and I have done sth similar) is to prepare a dict that would look like:</p>
<blockquote>
<p>dict = {‘model1’: Model_1, ‘model2’: Model_2, … }</p>
</blockquote>
<p>where Model_X is a PyTorch model with loaded weights and sitting on CPU.</p>
<p>During the run I would retrieve models sequentially by just getting one at a time from the dict. Move it to GPU, run inference, move back to CPU, clear GPU cache, load model2, …, etc.</p>
<p>The problem is definitely the time of loading/unloading models to/from GPUs. That would take a while definitely.</p>

WaterKnight: <p>I have find a better approach <a class="mention" href="/u/bonzogondo">@bonzogondo</a> :</p>
<aside class="quote" data-post="3" data-topic="83948">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/waterknight/48/54725_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/api-rest-with-several-models-loaded-using-gpu-but-not-at-same-time/83948/3">API Rest with several models loaded using GPU but not at same time</a> 
  </div>
  <blockquote>
    Thank you for comming to this issue <img width="20" height="20" src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=12" title="sweat_smile" alt="sweat_smile" class="emoji"> 
I have tried that approach, the problem that I have seen is that if I don’t delete pred variable after using, it keeps memory in use and when I load other model cuda memory error is thrown!
  </blockquote>
</aside>

<pre><code class="lang-auto">models = [...] # define a list of all models on the CPU

input = ... # get your input
for model in models:
    model.to('cuda')
    pred = model(input)
    #make something with pred
    del pred
    model.to('cpu')
</code></pre>
<p>They key fact is that you need to delete predictions variable if you don’t wank cuda memory issues</p>

gchauhan: <p>This will be a good use case for TorchServe which supports multiple models. Please give that a try and provide your feedback.</p>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="16" height="16">
      <a href="https://github.com/pytorch/serve" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars2.githubusercontent.com/u/21003710?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/pytorch/serve" target="_blank" rel="nofollow noopener">pytorch/serve</a></h3>

<p>Model Serving on PyTorch. Contribute to pytorch/serve development by creating an account on GitHub.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>