I want to know how to call the loader function of Pytorch in the Python/C API

tanik12: <p>I want to use Pytorch’s loader function using the Python/C API.<br>
The following steps (1) to (3) are used to process. (3) is the problem.</p>
<p>(1) Call a python file from C++.<br>
C++ file is a ConsoleApplication1.cpp.</p>
<p>(2) Load the loader function of pytorch from the python file.<br>
Python files are test1.py and DataSetSample.py.</p>
<p>(3) Repeated processing does not work. (Pytorch’s loader function doesn’t work properly.)<br>
Iteration in line 23 of test1.py doesn’t work. (line 24 works.)<br>
Curiously, up to line 22 of test1.py is infinitely processed.<br>
If you comment out the matplotlib in line 7 of test1.py, it stops at line 22.(test1.py)</p>
<p>I would like to know how to execute the Pytorch loader function in line 23.(test1.py)</p>
<h2><br></h2>
<pre><code class="lang-auto">Environment

◆OS
 Windows10
◆Environment related to C++
 C++ environment was created in Visual Studio Community 2017.
◆Environment related to python
 python   3.6.8
 torch       1.4.0
 torchvision 0.5.0
 matplotlib  3.2.1
</code></pre>
<h2><br></h2>
<p>This file is ConsoleApplication1.cpp.</p>
<pre><code class="lang-auto">#include &lt;stdio.h&gt;
#include &lt;Python.h&gt;
#include &lt;iostream&gt;


using namespace std;

int main() {
	PyObject *pName, *pModule, *pTmp, *pFunc;
	char *sTmp;
	int data;

	Py_Initialize();
	PyObject *sys = PyImport_ImportModule("sys");
	PyObject *path = PyObject_GetAttrString(sys, "path");
	PyList_Append(path, PyUnicode_DecodeFSDefault("executable file")); //Path of executable file
	pName = PyUnicode_DecodeFSDefault("test1");
	pModule = PyImport_Import(pName);
	Py_DECREF(pName);

	if (pModule != NULL) {
		pFunc = PyObject_GetAttrString(pModule, "test");
		pTmp = PyObject_CallObject(pFunc, NULL);
		int pp = 0;
		cin &gt;&gt; pp;

	}
	Py_Finalize();

	return 0;
}
</code></pre>
<hr>
<p>This file is  test1.py</p>
<pre><code class="lang-auto"># coding: UTF-8

print("call test1")
import torch

import DataSetSample
import matplotlib # this must affect the loop 

#### data loader ####
# Data
print('==&gt; Preparing data..')
dataset = DataSetSample.DataLoad()
loader = torch.utils.data.DataLoader(
    dataset,
    batch_size=5,
    num_workers=1,
    shuffle=True
)
#########################

def test():
    print("========================= forの前")
    for batch_idx, (_, _, _) in enumerate(loader): #can't enter loop
    #for _ in range(5):                            #enter loop
        print("test")

if __name__ == "__main__":
    test()
</code></pre>
<hr>
<p>This file is  DataSetSample.py</p>
<pre><code class="lang-auto">class DataLoad():
    def __init__(self):
        print("start")

    def __getitem__(self, index):
        return 1, 2, 3
    
    def __len__(self):
        return 3
</code></pre>
<br>
The result of running the executable after compiling ConsoleApplication1.cpp.
<pre><code class="lang-auto">call test1
==&gt; Preparing data..
start
========================= forの前
call test1
==&gt; Preparing data..
start
========================= forの前
call test1
==&gt; Preparing data..
start
========================= forの前
call test1
==&gt; Preparing data..
start
========================= forの前

Repeated processing...
</code></pre>
<p>Please help me.<br>
Thank you for reading to the end.</p>

ptrblck: <p>I’m not familiar with this particular use case of executing a Python application from C++, but could you set <code>num_workers=0</code> and retry it again, as you might be running into multiprocessing issues?</p>

tanik12: <p>Thank you for your response.<br>
It was really helpful.</p>
<p>I tried reducing num_workers to 1 at the time, but I couldn’t do it.<br>
I followed the advice and set num_workers to 0 and it worked!</p>

Edoar: <p>Hi! i’m trying to do something as you are, calling a python script that uses torch from a C++ app. I’m having problems when adding the “import torch” line to the py script, seems like i am not working in the right py environment and that there is no torch installed in that one. How did you assure that the py environment that you are working in has torch/matplotlib ecc ecc? Thanks!</p>