Couple of models in production

username: roaffix
cooked: <p>Hi everyone,</p>
<p>I’m trying to use couple of different models simultaneously in one big algorithm. Is it possible to run them on single GPU at the same time? And as I see, its kinda hard to deploy pytorch models in production pipeline. Should I turn to ONNX+Caffe2?</p>
<p>Thanks,<br>
Anton</p>

username: ptrblck
cooked: <p>It should be possible to run different models on the same GPU, however I think you could lose a lot of performance, since the models would have to wait for each other to finish the processing.</p>
<p>Maybe <a href="https://pytorch.org/docs/master/notes/multiprocessing.html#best-practices-and-tips">multiprocessing</a> might help, but I’m not really familiar with all the limitations.</p>
<p>What kind of deployment environment do you have?<br>
You could easily setup a webserver using flask or any other framework and serve your models there.<br>
If you need a lot of throughput on a local machine, I would go for ONNX and Caffe2.</p>
<p>PyTorch 1.0 will support easy deployment with Caffe2 as stated <a href="https://code.facebook.com/posts/172423326753505/">here</a>. You would have to wait a few months though, because the version is scheduled to be released this summer/autumn as far as I know.</p>

username: roaffix
cooked: <blockquote>
<p>however I think you could lose a lot of performance, since the models would have to wait for each other to finish the processing.</p>
</blockquote>
<blockquote>
<p>What kind of deployment environment do you have?</p>
</blockquote>
<p>I used a local machine and it was OK for other frameworks. E.g., in Tensorflow I created two different sessions and used them in the “production pipeline”. So each session was used individually and they were manually allocated in memory. Is it possible to make such a trick with pytorch or onnx + caffe2?</p>
<blockquote>
<p>PyTorch 1.0 will support easy deployment with Caffe2 as stated here.</p>
</blockquote>
<p>As I understood, release 1.0 will just simplify the onnx+caffe2 conversion. Am I wrong and it will be another solution? Anyway, I would like to find an alternative 1.0 solution now, if it possible <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="roaffix" data-post="3" data-topic="18501">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/roaffix/48/4611_2.png" class="avatar"> roaffix:</div>
<blockquote>
<p>I used a local machine and it was OK for other frameworks. E.g., in Tensorflow I created two different sessions and used them in the “production pipeline”. So each session was used individually and they were manually allocated in memory. Is it possible to make such a trick with pytorch or onnx + caffe2?</p>
</blockquote>
</aside>
<p>You could just create several different models, push them onto the GPU and feed your data.<br>
I suppose Tensorflow is doing the same.<br>
However, since the GPU has limited resources, your performance might be limited, since the models might have to wait for each other.<br>
You could try to use the CPU instead, if you have single input images for example.</p>
<p>As a side note: maybe <a href="https://github.com/pytorch/glow">glow</a> might be interesting for you.</p>
<aside class="quote no-group" data-username="roaffix" data-post="3" data-topic="18501">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/roaffix/48/4611_2.png" class="avatar"> roaffix:</div>
<blockquote>
<p>As I understood, release 1.0 will just simplify the onnx+caffe2 conversion. Am I wrong and it will be another solution? Anyway, I would like to find an alternative 1.0 solution now, if it possible <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
</blockquote>
</aside>
<p>Yeah, that’s also how I understand it.</p>

username: roaffix
cooked: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="18501">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>You could just create several different models, push them onto the GPU and feed your data.</p>
</blockquote>
</aside>
<p>So, as I understood:</p>
<ol>
<li>I save a model this way</li>
</ol>
<pre data-code-wrap="python"><code class="lang-python"># ... some code here
torch.save(model.state_dict(), "{}.pt".format(output_name))
# NOTE: output_name is modelX_name in next steps
</code></pre>
<ol start="2">
<li>Then I load model this way</li>
</ol>
<pre data-code-wrap="python"><code class="lang-python">model1 = ... # your model1. E.g, model1 = CatDogClassifier().to('cuda')
model2 = ... # your model2
model1.load_state_dict(torch.load(model1_name))
model2.load_state_dict(torch.load(model2_name))
</code></pre>
<ol start="3">
<li>And If I want to make a prediction I simply type</li>
</ol>
<pre data-code-wrap="python"><code class="lang-python">X = ... #some data. 
# e.g, X = torch.tensor(X, requires_grad=False, dtype=torch.float).to('cuda')
out1 = torch.max(model1(X), 1)[1]
# and for numpy output if you use 'cuda':
# out1 = torch.max(model1(X), 1)[1].cpu().numpy()
</code></pre>
<p>Please, correct me if I’m wrong. Can you provide something like pseudocode if I’m missed something?</p>

username: ptrblck
cooked: <p>Looks perfectly fine!<br>
Your model might not be located at <code>torch.Model()</code>, but I assume that’s just a typo.</p>
<p>You should definitely check, if CPU won’t be faster for single inputs.</p>

username: roaffix
cooked: <p>Thanks. I will try ASAP</p>

username: roaffix
cooked: <p>So, in my typo <code>model1 = torch.Model()</code> I loaded a model as the class. Is it possible to avoid this step?<br>
Also, <code>state_dict</code> raises the error with unexpected and missing keys.</p>

username: ptrblck
cooked: <p>I’m not sure, how else you would like to get your predictions.<br>
A model might be the easiest solution.<br>
Why do you want to avoid it?</p>
<p>The <code>state_dict</code> errors is thrown, if you save a model and change its architecture after.</p>

username: roaffix
cooked: <p>Just tried to avoid long dependencies.</p>
<p>upd. Solved problem with <code>state_dict</code>. That was my mistake <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: roaffix
cooked: <p><em><strong>Note for hackers, who will look through this post later.</strong></em> I changed a little bit post with a solution, so don’t get confused about later my or <a class="mention" href="/u/ptrblck">@ptrblck</a> comments <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=5" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: SixerWang
cooked: <p>I am sorry, I don’t understand how to run two models at the same time, your code could do this?</p>