Torch.compile's deployment story to non-Python host processes

username: vadimkantorov
cooked: <p>What is current <code>torch.compile</code>’s deployment story to non-python host processes?</p>
<p>TorchScript/TorchJit used to have <code>torchcpp</code>’s <code>torch::jit::load</code> and Python-free interpreting machinery</p>
<ol>
<li>
<p>Is there currently such a GIL-free graph/code interpreter?</p>
</li>
<li>
<p>Is <code>torchdeploy</code> supposed to help with that?</p>
</li>
<li>
<p>In general, how are TorchDynamo graphs exported? Can we first compile / generate the kernels and then deploy the kernels?</p>
</li>
</ol>
<p>It seems that vanilla cPython does not support sub-interpreters yet (waiting for PEP 684 implementation in Python 3.12) and even if the interpreter itself supports them, it’s unclear how NumPy / PyTorch would survive them: <a href="https://discuss.python.org/t/pep-684-a-per-interpreter-gil/19583/2" class="inline-onebox" rel="noopener nofollow ugc">PEP 684: A Per-Interpreter GIL - #2 by guido - PEPs - Discussions on Python.org</a> (if they use global state / registrations / allocator structures)</p>
<p>So having some simple threadsafe GIL-free Python-like interpreters is useful. What is the current PyTorch story for this usecase?</p>

username: marksaroufim
cooked: <p>It’s going to be <code>torch._dynamo.export()</code> and <code>torch._inductor.aot_compile()</code> for environments without and with Triton respectively but they’re both not ready yet but you can still poke up around and see what’s up there.</p>

username: vadimkantorov
cooked: <p>How are these “exported” bits are supposed to be used? Are these supported to run within some Python env? Or some libtorch-based Python-free interpreter for exported bits will exist? What artifact types are <code>torch._dynamo.export()</code> and <code>torch._inductor.aot_compile()</code> producing?</p>
<p>It seems that sub-interpreters are finally getting implemented really soon in stock Python: <a href="https://github.com/pytorch/pytorch/issues/102517" class="inline-onebox" rel="noopener nofollow ugc">[feature request] PyTorch support for sub-interpreters with PEP 684 accepted and release in Python 3.12 · Issue #102517 · pytorch/pytorch · GitHub</a> in PEP 684.</p>
<p>I also wonder if torch::deploy / torch::package are related to this hosting scenario?</p>

username: marksaroufim
cooked: <p>So for both of these the expectation is they’d run without Python but should be able to pybind them to do run them. I’m actually spending time this week trying to document the usability gaps here so will keep you posted</p>
<p>Regarding <code>torch.deploy</code> I can’t comment on it too much since the repo hasn’t been getting too much activity lately <a href="https://github.com/pytorch/multipy" class="inline-onebox" rel="noopener nofollow ugc">GitHub - pytorch/multipy: torch::deploy (multipy for non-torch uses) is a system that lets you get around the GIL problem by running multiple Python interpreters in a single C++ process.</a></p>

username: vadimkantorov
cooked: <p>There is also a question of multi-threading regarding PyTorch interpretator instances (given that until full support of sub-interpreters in Python/NumPy/PyTorch and without multipy this is very not obvious) - should be clearly stated if it’s supported or not and some best practices if it’s supported (e.g. to share the loaded model in memory between all threads/forked processes).</p>
<p>Sometimes it might be important: e.g. when we have several models and complex pipelining</p>