TorchScript model loading guidance

iamexperimentingnow: <p>I have a question, is that something we don’t want to initialize the model architecture and load the weights to do inference, if I save the model in Torch script (TorchScriptModule)format.</p>
<p>In the following below script, they are loading the save traced model directly. So, does it mean that we don’t want to initialize the model and load the saved weight?</p>
<pre><code class="lang-auto">import torch
import torchvision

# An instance of your model.
model = torchvision.models.resnet18()

# An example input you would normally provide to your model's forward() method.
example = torch.rand(1, 3, 224, 224)

# Use torch.jit.trace to generate a torch.jit.ScriptModule via tracing.
traced_script_module = torch.jit.trace(model, example)

traced_script_module.save("traced_resnet_model.pt")

loaded = torch.jit.load('traced_resnet_model.pt')
</code></pre>
<p>However, traditional way for loading the saved weights is to, first initialize the model and load the saved weights like the below steps</p>
<pre><code class="lang-auto"># Define model
class TheModelClass(nn.Module):
    def __init__(self):
        super(TheModelClass, self).__init__()
        self.conv1 = nn.Conv2d(3, 6, 5)
        self.pool = nn.MaxPool2d(2, 2)
        self.conv2 = nn.Conv2d(6, 16, 5)
        self.fc1 = nn.Linear(16 * 5 * 5, 120)
        self.fc2 = nn.Linear(120, 84)
        self.fc3 = nn.Linear(84, 10)

    def forward(self, x):
        x = self.pool(F.relu(self.conv1(x)))
        x = self.pool(F.relu(self.conv2(x)))
        x = x.view(-1, 16 * 5 * 5)
        x = F.relu(self.fc1(x))
        x = F.relu(self.fc2(x))
        x = self.fc3(x)
        return x

# Initialize model
model = TheModelClass()

# Initialize optimizer
optimizer = optim.SGD(model.parameters(), lr=0.001, momentum=0.9)

torch.save(model.state_dict(), PATH)

model = TheModelClass(*args, **kwargs)
model.load_state_dict(torch.load(PATH))
model.eval()
</code></pre>
<p>can someone please help me understand this?</p>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a> <a class="hashtag" href="/c/vision/5">#<span>vision</span></a> <span class="hashtag">#torchscript</span> <a class="hashtag" href="/c/deployment/14">#<span>deployment</span></a> <a class="hashtag" href="/c/jit/13">#<span>jit</span></a></p>

ptrblck: <aside class="quote no-group" data-username="iamexperimentingnow" data-post="1" data-topic="154312">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iamexperimentingnow/48/43455_2.png" class="avatar"> iamexperimentingnow:</div>
<blockquote>
<p>In the following below script, they are loading the save traced model directly. So, does it mean that we don’t want to initialize the model and load the saved weight?</p>
</blockquote>
</aside>
<p>Yes, that’s right. You can load the scripted or traced model directly (e.g. as is done if you want to execute this model in <code>libtorch</code> after exporting it in the Python API).</p>

iamexperimentingnow: <p>thanks for your reply <a class="mention" href="/u/ptrblck">@ptrblck</a> , here I believe libtorch is C++ packages. I’m afraid I’m not a C++ developer.<br>
is it mandatory to load the model using “libtorch” and do the inference?</p>
<p>I have a follow-up question,</p>
<ol>
<li>only if I load the model using libtorch and do the inference, then I see the time difference during inferencing?<br>
or</li>
<li>is it possible to load the model in python and do the inference? will that reduce the inference time?</li>
</ol>

ptrblck: <aside class="quote no-group" data-username="iamexperimentingnow" data-post="3" data-topic="154312">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iamexperimentingnow/48/43455_2.png" class="avatar"> iamexperimentingnow:</div>
<blockquote>
<p>is it mandatory to load the model using “libtorch” and do the inference?</p>
</blockquote>
</aside>
<p>No, you can also deploy the workload using the Python frontend.</p>

iamexperimentingnow: <p>is it a traditional way, like loading the  torchscript model using</p>
<pre><code class="lang-auto">model = torch.jit.load('model_scripted.pt')
model.eval()
</code></pre>
<p>and do the inference.</p>
<p>if I do by this way will it reduce the inference timing?</p>
<p>because I tested it with an example provided in the pytorch official documentation<br>
where they have used resnet50 model. I tried with that, by saving the model in a traditional way and torchscript way as well. After some warmup iterartion, I didn’t see any difference in the inference timing.</p>
<p>could you please help here?</p>

ptrblck: <p>Scripting the model can provide a speedup but also depends on a lot of factors such as the used JIT backend (related to the used PyTorch version) etc.<br>
As mentioned in the other post, you could try out the current nightly to see if scripting the model would yield a speedup using the new features in the <code>nvfuser</code> backend we were working on.</p>

iamexperimentingnow: <aside class="quote no-group" data-username="ptrblck" data-post="6" data-topic="154312">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>you could try out the current nightly to see</p>
</blockquote>
</aside>
<p>thanks for your reply <a class="mention" href="/u/ptrblck">@ptrblck</a> , could you please tell me to which post/article you’re referring to?</p>
<p>could you please send me an example article/blog to emulate the same and see the difference?<br>
<a class="mention" href="/u/ptrblck">@ptrblck</a></p>

ptrblck: <aside class="quote no-group" data-username="iamexperimentingnow" data-post="7" data-topic="154312">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iamexperimentingnow/48/43455_2.png" class="avatar"> iamexperimentingnow:</div>
<blockquote>
<p>could you please tell me to which post/article you’re referring to?</p>
</blockquote>
</aside>
<p>Sure, I was referring to your <a href="https://discuss.pytorch.org/t/got-stuck-in-torchscript-trace/154353/2">other post</a> where I’ve mentioned the same.</p>
<aside class="quote no-group" data-username="iamexperimentingnow" data-post="7" data-topic="154312">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iamexperimentingnow/48/43455_2.png" class="avatar"> iamexperimentingnow:</div>
<blockquote>
<p>could you please send me an example article/blog to emulate the same and see the difference?</p>
</blockquote>
</aside>
<p>Once the release is done, I can forward you the article(s).</p>

iamexperimentingnow: <p>thanks very much for your reply, <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>