How to serialize models with torch.compile properly

snakers41: <p>Hi,</p>
<p>Despite the main points in the <code>torch.compile</code> pitch, we faced some issues with <code>jit</code>, but they were tolerable, and we adopted <code>torch.jit.save</code> and torch packages as a model serialization / obfuscation / freezing methods (and ONNX as well).</p>
<p>It may be seen as a disadvantage, but sharing single <code>.jit</code> or <code>.package</code> artefacts may be preferrable to sharing the whole model codebase and then running <code>torch.compile</code>.</p>
<p>So, I have a few general questions, which I believe were not yet clearly anwered in the docs / blog / release materials:</p>
<ul>
<li>What is the preferred serialization format for <code>torch.compile</code>-ed models?</li>
<li>How does JIT compiler, <code>.jit</code> format, torch packages play with compiled models?</li>
<li>Should I first compile the model, then export to <code>jit</code> or <code>package</code>, or vice versa, or am I getting it wrong altogether?</li>
</ul>
<p>Or maybe it is still early to ask these questions? Maybe there are some discussions? Found <a href="https://dev-discuss.pytorch.org/t/what-is-the-recommend-serialization-format-when-considering-the-upcoming-pt2/1088" rel="noopener nofollow ugc">this</a>, but it seems not very informative.</p>
<p>PS<br>
Also forgive me for a dumb question, but why there are now 2 official PyTorch forums, this one, and <a href="http://dev-discuss.pytorch.org" rel="noopener nofollow ugc">dev-discuss.pytorch.org</a>, the latter being ~100x smaller?</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/8/485d36af4fc49bc4fef4f43b7f2fe497e436c5ee.png" data-download-href="https://discuss.pytorch.org/uploads/default/485d36af4fc49bc4fef4f43b7f2fe497e436c5ee" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/485d36af4fc49bc4fef4f43b7f2fe497e436c5ee_2_345x188.png" alt="image" data-base62-sha1="aka5oSdA0SpH17gKWbxIBQHj1gO" width="345" height="188" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/485d36af4fc49bc4fef4f43b7f2fe497e436c5ee_2_345x188.png, https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/485d36af4fc49bc4fef4f43b7f2fe497e436c5ee_2_517x282.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/4/8/485d36af4fc49bc4fef4f43b7f2fe497e436c5ee_2_690x376.png 2x" data-dominant-color="252525"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1047×571 20.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/e/eed951aae0853f2eb324f9d7bffbae738bf7ec4a.png" data-download-href="https://discuss.pytorch.org/uploads/default/eed951aae0853f2eb324f9d7bffbae738bf7ec4a" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/eed951aae0853f2eb324f9d7bffbae738bf7ec4a_2_345x178.png" alt="image" data-base62-sha1="y4XeZvc4c9mG8ZaSj3w1pXiznDk" width="345" height="178" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/eed951aae0853f2eb324f9d7bffbae738bf7ec4a_2_345x178.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/eed951aae0853f2eb324f9d7bffbae738bf7ec4a_2_517x267.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/e/eed951aae0853f2eb324f9d7bffbae738bf7ec4a_2_690x356.png 2x" data-dominant-color="FBFBFB"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1059×549 16.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

marksaroufim: <p>There is no serialization solution yet for <code>torch.compile</code> but it’s high priority</p>
<p>Regarding the 2 forums this is more of a community forum for general pytorch users wheras the dev forum is more for pytorch contributors. IMO this place feels more like a forum and dev discuss is closer to a blog</p>

snakers41: <p>I see, many thanks for your reply.</p>

snakers41: <p>Hi again!</p>
<p>Since PyTorch 2.1 was released yesterday, I immediately rushed to to read the release notes, and still there is not much information about proper packaging.</p>
<p>Am I missing something, or the compile being still in BETA, packaging is still to be solved in future?</p>
<p>Best,<br>
Alexander</p>

snakers41: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/f/7f0800bc7510c8a4d4b8a81a01e82a68a1ba2add.png" data-download-href="https://discuss.pytorch.org/uploads/default/7f0800bc7510c8a4d4b8a81a01e82a68a1ba2add" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7f0800bc7510c8a4d4b8a81a01e82a68a1ba2add_2_690x240.png" alt="image" data-base62-sha1="i7LMESIRkBNKR5gjYJjQJUXcmoB" width="690" height="240" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7f0800bc7510c8a4d4b8a81a01e82a68a1ba2add_2_690x240.png, https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7f0800bc7510c8a4d4b8a81a01e82a68a1ba2add_2_1035x360.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/7/f/7f0800bc7510c8a4d4b8a81a01e82a68a1ba2add_2_1380x480.png 2x" data-dominant-color="FAFAFA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1717×599 50.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>