Calling forward on TorchScript model multiple times leads to error

will-leeson: <p>I’m using a TorchScript model to make decisions in my C++ code. To try to make it more efficient, I made a model class, which can load the model weights and be called to make predictions. This allows me to load the model a single time instead of every time I want to call forward. Since I plan on calling the model many times, this can lead to significant savings.</p>
<p>The model successfully calls forward twice, but on the third call, I receive the following error:<br>
<code>Unable to find target for this triple (no targets are registered)[1]    38685 IOT instruction (core dumped)</code><br>
I’m not sure what is calling this since it is able to call forward twice. If I create the model every time I want to call forward, it works fine. Does anyone have any insight? If it helps, here is my model class. Note that the model is a PyTorch-Geometric model, I don’t think that will make a difference though.</p>
<pre><code class="lang-auto">gat::gat(std::string path){
    try{
        model = torch::jit::load(path);
        loaded = true;
    }
    catch(const c10::Error &amp;e){
        loaded = false;
    }
}

int gat::predict(std::vector&lt;unsigned int&gt; nodes, 
                 std::vector&lt;unsigned int&gt; inEdges,
                 std::vector&lt;unsigned int&gt; outEdges){

    //Need to declare it as int type to avoid conversion issues
    auto opts = torch::TensorOptions().dtype(torch::kInt32);
    //Need to convert it to a Float for the model
    auto nodeTensor = torch::from_blob(nodes.data(), {(unsigned int)nodes.size()/67,67}, opts).to(torch::kFloat32);

    auto inEdgeTensor = torch::from_blob(inEdges.data(), (unsigned int)inEdges.size(), opts).to(torch::kI64);
    auto outEdgeTensor = torch::from_blob(outEdges.data(), (unsigned int)outEdges.size(), opts).to(torch::kI64);
    auto edgeTensor = torch::stack({inEdgeTensor, outEdgeTensor});

    auto edge_attrTensor = torch::from_blob(edge_attr.data(), edge_attr.size(), opts).to(torch::kFloat32);

    std::vector&lt;torch::jit::IValue&gt; inputs;
    inputs.push_back(nodeTensor);
    inputs.push_back(edgeTensor);

    auto out = model.forward(inputs).toTensor();
    int choice = out.argmin().item&lt;int&gt;();

    return choice;
}
</code></pre>

Stefan_Heinrich: <p>I got the same issue<br>
Are there any solutions?</p>

oopsno: <p>+1, got a similar, but more dramatic issue.</p>
<p>I’m trying to trace the <code>swin-v2-tiny</code> model with following code:</p>
<pre><code class="lang-auto">import torch
from torchvision.io.image import read_image
from torchvision.models.swin_transformer import (
    swin_v2_t,
    Swin_V2_T_Weights as W
)

weights = W.IMAGENET1K_V1  # 28,351,570  params
preprocess = weights.transforms()
model = swin_v2_t(weights=weights).eval()
image = read_image("images/coco-178692.jpg")
batch = preprocess(image).unsqueeze(0)

script = torch.jit.trace(model.forward, batch)

for _ in range(1000):
    script(batch)

torch.jit.save(script, 'swin.pth')
</code></pre>
<p>These code works as expected. However, if I add</p>
<pre><code class="lang-auto">import rvm.relay.op
</code></pre>
<p>right <strong>before</strong> <code>import torch</code>, my code will crash at <code>script(batch)</code> with</p>
<pre><code class="lang-auto">Program aborted due to an unhandled Error:
Unable to find target for this triple (no targets are registered)[1]    4129529 abort (core dumped)  python3.8 swinv2.py
</code></pre>
<p><code>import tvm.realy.op</code> at anywhere <strong>after</strong> <code>import torch</code> is fine.</p>
<p>My env:</p>
<ul>
<li>Intel 10700K + RTX3070</li>
<li>Ubuntu 20.04</li>
<li>PyTorch 1.13.0+cu117 with Python 3.8</li>
<li>TVM 0.11dev~<a href="https://github.com/apache/tvm/tree/3a81aef40bca9479d4a691b3a80e42b01f3f8a0d" rel="noopener nofollow ugc">3a81aef</a>
</li>
</ul>