How to Unload model in C++?

a_bear_paw: <p>Hi,<br>
I need to unload model which loaded by torch::jit::load() with C++. But I couldn’t find anything about it just like ‘torch::jit::unload()’.</p>
<ol>
<li>In my C++ code, there are two functions and provided api using pybind11:</li>
</ol>
<pre><code class="lang-auto">#include &lt;iostream&gt;
#include &lt;vector&gt;
#include &lt;string&gt;

#include &lt;torch/torch.h&gt;
#include &lt;torch/script.h&gt;

#include &lt;opencv2/core.hpp&gt;
#include &lt;opencv2/imgcodecs.hpp&gt;
#include &lt;opencv2/highgui/highgui.hpp&gt;
#include &lt;opencv2/imgproc.hpp&gt;
...
torch::jit::script::Module initialize(const std::string &amp;fpath) {
    torch::jit::script::Module model = torch::jit::load(fpath, "cuda");
    model.to("cuda:0");
    return model;
}


std::tuple&lt;at::Tensor, at::Tensor&gt; do_inference(torch::jit::scriot::Module model, ...) {
    ...  // pre-processing
    at::Tensor out = model.forward(input).toTensor();
    ...  // post processing
}

PYBIND11_MODULE(...) {
    m.def("initialize", &amp;initialize, ...)
    ...
    m.def("do_inference",  &amp;do_inference, ...)
}
</code></pre>
<ol start="2">
<li>Build it to foo.so, and import this module in Python3.6, python is a web service, just like:</li>
</ol>
<pre><code class="lang-auto">import foo
from flask import Flask  # a web service
...

model = foo.initialize("./resnet50.pt")

img = open("./kitty.jpg", "rb")
foo.doInference(img)

# This service is running
</code></pre>
<ol start="3">
<li>The Web service is working fine, but how to unload model from device(GPU or CPU), If I want to update the model without stopping the service.<br>
Is there a solution for that out there?<br>
Thanks!</li>
</ol>

ptrblck: <p>Could you just <code>del model</code> and clear the cache?<br>
I don’t see what any modules, references etc. are stored in the C++ code, so I assume the objects can be freed in your Python script.</p>

a_bear_paw: <p>Yes, I want to del model and clear the cache, but NOT shutdown the Python process.<br>
In fact, C++ code is used for inference(including pre-processing and post-processing), The Python process(which is running as a service) load model and inference by calling the C++ dynamic library(may have multiple, one dynamic library, one model).<br>
So, Is there a way to delete loaded model without stopping the Python process?<br>
Thanks!</p>

ptrblck: <p>Yes, you can delete the model by running <code>del model</code> in the Python script without shutting it down.</p>

a_bear_paw: <p>I’m trying to use del model in the Python process, but it seem doesn’t work.<br>
when I using nvidia-smi, the model process is still here and the GPU memory not be released.<br>
It seems that the model is loaded in C++ context and must be released in the C++, right?</p>

ptrblck: <p>The memory might still be in the cache and thus not released. You could run <code>torch.cuda.empty_cache()</code> to get it back. I’m not 100% sure, but don’t think you need to release it in the C++ backend, since you are holding a reference to it in Python.</p>

a_bear_paw: <p>I use <code>torch.cuda.memory_summary("cuda")</code> before and after the <code>del model</code>, I found the “Allocated memory” to be released after <code>del model</code>, It seems worked. Thanks.</p>