How are python bindings created?

PranovD: <p>I’m trying to figure out how the python bindings are generated as I see that pybind is used in some cases but it also seems like some python bindings are generated by custom files: <a href="https://github.com/pytorch/pytorch/blob/master/tools/setup_helpers/generate_code.py" rel="nofollow noopener">https://github.com/pytorch/pytorch/blob/master/tools/setup_helpers/generate_code.py</a></p>
<p>Understanding how and why this works will be super helpful to me as I am trying to create C# bindings for libtorch.</p>

colesbury: <p>Most of the core PyTorch functions on the <code>torch</code> package and on <code>torch.Tensor</code> are generated from Declarations.yaml (*) using <a href="https://github.com/pytorch/pytorch/blob/master/tools/autograd/gen_python_functions.py" rel="nofollow noopener">gen_python_functions.cpp</a>. The generated files look like:</p>
<p><a href="https://gist.github.com/colesbury/9d93a8bcf72acabdd77c9ca805c6534c" rel="nofollow noopener">python_torch_functions.cpp</a><br>
<a href="https://gist.github.com/973938719e613973aee1464f0bc7900e" rel="nofollow noopener">python_variable_methods.cpp</a></p>
<p>We don’t use pybind11 here for a few reasons:</p>
<ol>
<li>We need to disambiguate multiple overloads in a PyTorch-specific way. Some of this behavior is due to historical Lua Torch function signatures. We try to avoid this ambiguity in new functions.</li>
<li>pybind11 tends to slow down compile time a lot</li>
</ol>
<p>(*) <a href="https://gist.github.com/a6a21d2cf40815258d4565f2373cf7d8" rel="nofollow noopener">Declarations.yaml</a> is generated from <a href="https://github.com/pytorch/pytorch/blob/master/aten/src/ATen/native/native_functions.yaml" rel="nofollow noopener">native_functions.yaml</a>. There used to be other inputs that went into Declarations.yaml, but I’m not sure if that’s still the case. Anyways, Declarations.yaml is intended as the “public” machine-readable API specification for things like language bindings.</p>

PranovD: <p>Because the bindings I am trying to make are in C# which supports method overloading, and compile time is not one of my constraints, would using a 3rd party bindings generator (similar to pybind) be recommended or are there other considerations?</p>

colesbury: <p>I don’t understand. How would pybind help with C#? Aren’t you trying to bind C++ functions in C#? (pybind11 binds C++ functions to Python)</p>

PranovD: <p>I would not be using pybind I would be using a generator that would be for C# similar to the way pybind is for python, something like <a href="http://www.swig.org/tutorial.html" rel="nofollow noopener">http://www.swig.org/tutorial.html</a></p>

adampauls: <p>I see that Declarations.yaml is generated when building pytorch. Ideally, when auto-generating bindings for libtorch, I wouldn’t have to run the complete build. Instead, I’d like to generate the Declarations.yaml file and then download the pre-built packages from <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a>. Is it possible for me to only run the build step that generates Declarations.yaml given a download of pytorch source?</p>
<p>Thanks!</p>

adampauls: <p>It seems that running <code>python -m tools.codegen.gen -s aten/src/ATen -d torch/share/ATen</code> from root of a source checkout puts Declarations.yaml in the expected place, though I’m not sure if it inappropriately puts other stuff in that folder as well.</p>