Torch.load(model) hangs indefinately when invoked via python bindings

Alex_john: <p>Hello,</p>
<p>When trying to invoke the pytorch inference code from c++ using python binding the code gets hung indefinitely in torch.load(model, map_location=“cuda:0”) for GPU models. However for cpu models I have no issue.</p>
<p>Although the GPU model hangs if invoked via python -bindings, the same inference code run successfully for both GPU and CPU models when invoked via python interpreter.</p>
<p>Following is the detail of my setup:<br>
Installation command: conda install pytorch torchvision cudatoolkit=10.2 -c pytorch<br>
nvcc version:<br>
nvcc: NVIDIA (R) Cuda compiler driver<br>
Copyright (c) 2005-2019 NVIDIA Corporation<br>
Built on Wed_Oct_23_19:24:38_PDT_2019<br>
Cuda compilation tools, release 10.2, V10.2.89</p>
<p>nvidia-smi:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/2/82f759e7d932ed8f1129151773b3ebe8e9d76400.png" data-download-href="https://discuss.pytorch.org/uploads/default/82f759e7d932ed8f1129151773b3ebe8e9d76400" title="Screenshot from 2020-05-24 13-12-02"><img src="https://discuss.pytorch.org/uploads/default/original/3X/8/2/82f759e7d932ed8f1129151773b3ebe8e9d76400.png" alt="Screenshot from 2020-05-24 13-12-02" data-base62-sha1="iGA19GbWFD9ZW98JdWBElDsXLNu" width="690" height="314" data-dominant-color="3E1A33"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot from 2020-05-24 13-12-02</span><span class="informations">727×331 35.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Any clue?</p>

ptrblck: <p>What kind of Python binding are you using?<br>
Could you post the C++ code, which creates this issue and if possible, how you’ve exported the model in Python?</p>

Alex_john: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>, thanks for the reply.</p>
<p>Following is the sample code that loads the pytorch code  “pytorch_inference.py”  and calls inference function inference_code(data_from_c++) from c++:</p>
<pre><code class="lang-auto">#include &lt;Python&gt;

//============Pytorch File loading=============================

    Py_Initialize();
    PyRun_SimpleString("import sys\nsys.path.append('./pytorch_dir')"); //path to dir where my pytorch code  pytorch_inference.py resides
    PyObject * pName_pytorch_file = PyUnicode_FromString("pytorch_inference"); //load pytorch_inference.py
    PyObject * pModule_pytorch_module = PyImport_Import(pName_pytorch_file);    
    if (pModule_pytorch_module != NULL) {
        PyObject * pFunc_pytorch_inference_pointer = PyObject_GetAttrString(pModule_pytorch_module, "inference_code"); //load the offset address of python function to be called from c++
		std::cout &lt;&lt; "[Python Func]: offset set" &lt;&lt; std::endl;       
    }
    else
    {
        std::cout &lt;&lt; "Module failed to load " &lt;&lt; std::endl;
        assert(0);
    }
//================call to specific function of python from c++====

if (pFunc_pytorch_inference_pointer &amp;&amp; PyCallable_Check(pFunc_pytorch_inference_pointer)) {	
	pArgs_to_pytorch_func = PyTuple_New(1);
    PyObject* data= PyArray_SimpleNewFromData(1,dims, PyArray_USHORT, bufferOfData);
	PyTuple_SetItem (pArgs_to_pytorch_func, 0, data);
    PyObject *Dict  = PyObject_CallObject(pFunc_pytorch_inference_pointer, data);  //this calls the pytorrch function "inference_code" with some data
}


//====================================================================
</code></pre>
<p>And the pytorch code pytorch_inference.py:</p>
<pre><code class="lang-auto">//====================Python side pytorch_inference.py ==========

print("import --endered")
import os
import pickle as pkl
import sys
import warnings
import shutil
import cv2
from PIL import Image
import numpy as np
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.nn.init
import torch.optim as optim
import torch.optim.lr_scheduler
import torch.utils.data as data
from torchvision.utils import save_image
import torchvision
print("import succedded-")
model_path = "./model_gpu.pth"

def inference_code(data_from_c++):
	dict_to_cpp = {}	
    torch.cuda.init()											#To test
    print("torch.cuda.is_available(): ",torch.cuda.is_available()) #To test: returned true
    print("torch.cuda.is_initialized(): ", torch.cuda.is_initialized())#To test: returned true
	torch.load(model_path,map_location="cuda:0")#----Hangs right here , but works for cpu models(for cpu I do not pass map_location="cuda:0")
	return dict
	
//===================================================================

</code></pre>
<blockquote>
<p>How you’ve exported the model in Python?</p>
</blockquote>
<p>I give the path to the model in the pytorch code itself:</p>
<pre><code class="lang-auto">model_path = "./model_gpu.pth"  #tried giving absolute path but still it hangs in torch.load
torch.load(model_path,map_location="cuda:0")

</code></pre>

ptrblck: <p>Is your script also hanging, if you use Python directly or just if you call it in C++ using the bindings?</p>
<p>Just out of curiosity, what’s your use case that you want to call Python from C++ instead of directly loading the model in C++?</p>

Alex_john: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="82669">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Is your script also hanging, if you use Python directly or just if you call it in C++ using the bindings?</p>
</blockquote>
</aside>
<p><a class="mention" href="/u/ptrblck">@ptrblck</a>, It hangs only when I call it from c++ and only for GPU models, CPU models runs fine even with the python bindings. If I call it from python it runs fine for both CPU and GPU models .</p>
<aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="82669">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Just out of curiosity, what’s your use case that you want to call Python from C++ instead of directly loading the model in C++?</p>
</blockquote>
</aside>
<p>we have few  post processing stuffs that are still evolving and needs to be done in python side itself for precipitation of design, and there are few other stuffs in c++ side that need not be implemented in python as they ought to be in c++. So, I just need the predictions from the python file in the form of dictionary, where I can use it in c++ to give the input to other module</p>

Alex_john: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Any clue on this? Lua based torch didn’t had any problem with gpu models when invoked from lua bindings, but the pytorch gets stuck in torch.load for GPU models when invoked via python bindings.</p>

ptrblck: <p>Unfortunately, I never called a Python script from C++, so I cannot be really helpful here. <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"><br>
A hang often comes from some multiprocessing issues / mismatches, so if possible, I would try to scale down the problem and use as few threads/processes as possible.</p>

Alex_john: <p><a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>I kinda figured out the problem. The thing is, whenever I load the pytorch inference code from cpu-main thread it works, but if I call the same from the cpu-worker thread it hangs in torch.load for all GPU models.</p>
<p>Do you know what is going on?</p>

RebirthT: <p><a class="mention" href="/u/alex_john">@Alex_john</a> any solution??<br>
Same problem here</p>

mike4: <p><a class="mention" href="/u/rebirtht">@RebirthT</a> <a class="mention" href="/u/alex_john">@Alex_john</a> Did any one manage to find a solution to this I have exactly the same issue</p>