Ubuntu C++ gpu load model error

RBoulderstone: <p>I created a torch script file from pretrained resnet 18 model in python:</p>
<p>class Image2VectorResnet18():<br>
def <strong>init</strong>(self, frameWidth, frameHeight, device):<br>
self.device = device<br>
self.model = ptcv_get_model(‘resnet18’, pretrained=True)<br>
self.model = self.model.to(self.device)<br>
self.model.eval()<br>
self.normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])</p>
<pre><code>def forward(self, image):
    image = self.normalize(self.toTensor(image)).unsqueeze(0).to(self.device)
    vector = self.model(image)
    return vector
</code></pre>
<h1><a name="h-1" class="anchor" href="#h-1"></a></h1>
<h1>
<a name="parameters-2" class="anchor" href="#parameters-2"></a>Parameters</h1>
<h1><a name="h-3" class="anchor" href="#h-3"></a></h1>
<p>frameWidth = 224<br>
frameHeight = 224</p>
<h1><a name="h-4" class="anchor" href="#h-4"></a></h1>
<h1>
<a name="select-cuda-as-device-5" class="anchor" href="#select-cuda-as-device-5"></a>select cuda as device</h1>
<h1><a name="h-6" class="anchor" href="#h-6"></a></h1>
<p>device = torch.device(“cuda”)</p>
<h1><a name="h-7" class="anchor" href="#h-7"></a></h1>
<h1>
<a name="initialise-model-8" class="anchor" href="#initialise-model-8"></a>initialise model</h1>
<h1><a name="h-9" class="anchor" href="#h-9"></a></h1>
<p>image2Vector = Image2VectorResnet18(frameWidth, frameHeight, device)<br>
model = image2Vector.model</p>
<h1><a name="h-10" class="anchor" href="#h-10"></a></h1>
<h1>
<a name="overwrite-output-stage-with-identity-11" class="anchor" href="#overwrite-output-stage-with-identity-11"></a>overwrite output stage with identity</h1>
<h1><a name="h-12" class="anchor" href="#h-12"></a></h1>
<p>model.output = torch.nn.Identity()<br>
newmodel = model</p>
<h1><a name="h-13" class="anchor" href="#h-13"></a></h1>
<h1>
<a name="create-dummy-input-14" class="anchor" href="#create-dummy-input-14"></a>create dummy input</h1>
<h1><a name="h-15" class="anchor" href="#h-15"></a></h1>
<p>image = torch.ones(1, 3, frameWidth, frameHeight).to(device)</p>
<h1><a name="h-16" class="anchor" href="#h-16"></a></h1>
<h1>
<a name="create-jit-file-17" class="anchor" href="#create-jit-file-17"></a>create jit file</h1>
<h1><a name="h-18" class="anchor" href="#h-18"></a></h1>
<p>traced_script_module = torch.jit.trace(newmodel, image)<br>
traced_script_module.save(‘resnet18.pt’)</p>
<p>When I try to load this in c++ with:</p>
<p>moduleFeatureVector = torch::jit::load(“resnet18.pt”);</p>
<p>Works if I use ‘cpu’ as device in Python, but if I use 'cuda, I get:</p>
<p>Could not run ‘aten::empty_strided’ with arguments from the ‘CUDA’ backend. This could be because the operator doesn’t exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit <a href="https://fburl.com/ptmfixes" class="inline-onebox" rel="noopener nofollow ugc">Internal Login</a> for possible resolutions. ‘aten::empty_strided’ is only available for these backends: [CPU, Meta, BackendSelect, Python, Named, Conjugate, Negative, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, AutogradLazy, AutogradXPU, AutogradMLC, AutogradHPU, AutogradNestedTensor, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, UNKNOWN_TENSOR_TYPE_ID, Autocast, Batched, VmapMode].</p>
<p>CPU: registered at aten/src/ATen/RegisterCPU.cpp:18433 [kernel]<br>
Meta: registered at aten/src/ATen/RegisterMeta.cpp:12703 [kernel]<br>
BackendSelect: registered at aten/src/ATen/RegisterBackendSelect.cpp:665 [kernel]<br>
Python: registered at …/aten/src/ATen/core/PythonFallbackKernel.cpp:47 [backend fallback]<br>
Named: registered at …/aten/src/ATen/core/NamedRegistrations.cpp:7 [backend fallback]<br>
Conjugate: fallthrough registered at …/aten/src/ATen/ConjugateFallback.cpp:22 [kernel]<br>
Negative: fallthrough registered at …/aten/src/ATen/native/NegateFallback.cpp:22 [kernel]<br>
ADInplaceOrView: fallthrough registered at …/aten/src/ATen/core/VariableFallbackKernel.cpp:64 [backend fallback]<br>
AutogradOther: registered at …/torch/csrc/autograd/generated/VariableType_2.cpp:10491 [autograd kernel]<br>
AutogradCPU: registered at …/torch/csrc/autograd/generated/VariableType_2.cpp:10491 [autograd kernel]<br>
AutogradCUDA: registered at …/torch/csrc/autograd/generated/VariableType_2.cpp:10491 [autograd kernel]<br>
AutogradXLA: registered at …/torch/csrc/autograd/generated/VariableType_2.cpp:10491 [autograd kernel]<br>
AutogradLazy: registered at …/torch/csrc/autograd/generated/VariableType_2.cpp:10491 [autograd kernel]<br>
AutogradXPU: registered at …/torch/csrc/autograd/generated/VariableType_2.cpp:10491 [autograd kernel]<br>
AutogradMLC: registered at …/torch/csrc/autograd/generated/VariableType_2.cpp:10491 [autograd kernel]<br>
AutogradHPU: registered at …/torch/csrc/autograd/generated/VariableType_2.cpp:10491 [autograd kernel]<br>
AutogradNestedTensor: registered at …/torch/csrc/autograd/generated/VariableType_2.cpp:10491 [autograd kernel]<br>
AutogradPrivateUse1: registered at …/torch/csrc/autograd/generated/VariableType_2.cpp:10491 [autograd kernel]<br>
AutogradPrivateUse2: registered at …/torch/csrc/autograd/generated/VariableType_2.cpp:10491 [autograd kernel]<br>
AutogradPrivateUse3: registered at …/torch/csrc/autograd/generated/VariableType_2.cpp:10491 [autograd kernel]<br>
Tracer: registered at …/torch/csrc/autograd/generated/TraceType_2.cpp:11425 [kernel]<br>
UNKNOWN_TENSOR_TYPE_ID: fallthrough registered at …/aten/src/ATen/autocast_mode.cpp:466 [backend fallback]<br>
Autocast: fallthrough registered at …/aten/src/ATen/autocast_mode.cpp:305 [backend fallback]<br>
Batched: registered at …/aten/src/ATen/BatchingRegistrations.cpp:1016 [backend fallback]<br>
VmapMode: fallthrough registered at …/aten/src/ATen/VmapModeRegistrations.cpp:33 [backend fallback]</p>

ptrblck: <p>Based on the error message it seems as if you are trying to load a traced model onto the GPU while your <code>libtorch</code> build doesn’t support the CUDA backend. Make sure to install the <code>libtorch</code> version built with CUDA or build it from source using CUDA.</p>