Using Python embedded interpreter via Pybind in C++ extension

almeetb: <p>Is it possible to use python embedded interpreter via pybind in a C++ extension? I want to invoke a python function in c++ but get an error using <code>py::module::import</code></p>
<p>My code looks like:</p>
<pre><code class="lang-auto">#include "torch/extension.h"

namespace py = pybind11;

void SomeFunction() {
  py::object python_func = py::module::import("python").attr("func")
  python_func();
}
</code></pre>
<p>I get the error:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/5/2528e5f849d4cf6066e4d2d74ff163608520a9be.png" data-download-href="https://discuss.pytorch.org/uploads/default/2528e5f849d4cf6066e4d2d74ff163608520a9be" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/2528e5f849d4cf6066e4d2d74ff163608520a9be_2_690x81.png" alt="image" data-base62-sha1="5iJi1gGrgW6YMR5PkA8Z8kb0tj0" width="690" height="81" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/2528e5f849d4cf6066e4d2d74ff163608520a9be_2_690x81.png, https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/2528e5f849d4cf6066e4d2d74ff163608520a9be_2_1035x121.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/2/5/2528e5f849d4cf6066e4d2d74ff163608520a9be_2_1380x162.png 2x" data-dominant-color="0E0E0E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2768×328 161 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Do I have to set pybind11::embed somewhere?</p>
<p>I am not able to import <code>torch/include/pybind11/embed.h</code> as this file cannot be found</p>

albanD: <p>Do you hold the GIL properly when you call these python constructs.<br>
Also did you name your module “python”? Maybe use a name less likely to be used in other places.</p>

almeetb: <p>I missed out on acquiring the GIL but now I get the same error on the call to <code>pybind11::gil_scoped_acquire gil</code></p>
<p>Am I missing a header in order to use pybind or do I need to start the interpreter by calling <code>py::scoped_interpreter guard{}</code> in my extension?</p>
<p>I didn’t name the module “python” that was just an example with some names changed.</p>

albanD: <p>You mean it sefault when you try to acquire the gil? Is python properly initialized alresady?</p>

almeetb: <p>Yes it segfaults on acquiring the GIL and python is properly initialized</p>

albanD: <p>This looks unrelated to pytorch. The whole stack trace points to pybind and cpython code…<br>
Could you give more information on how you build this?</p>

almeetb: <p>I built it using instructions from <a href="https://pytorch.org/tutorials/advanced/cpp_extension.html" rel="nofollow noopener">https://pytorch.org/tutorials/advanced/cpp_extension.html</a></p>
<p>But I was able to resolve this issue using the workaround in: <a href="https://github.com/pybind/pybind11/issues/1738" rel="nofollow noopener">https://github.com/pybind/pybind11/issues/1738</a></p>

albanD: <p>Ho interesting. It does look like an initialization problem then <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>