Torch-MLIR - Bridging PyTorch and MLIR ecosystems

powderluv: <p>The Torch-MLIR project aims to provide first class compiler support from the <a href="https://pytorch.org/" rel="noopener nofollow ugc">PyTorch</a> ecosystem to the MLIR ecosystem.</p>
<p><a href="https://mlir.llvm.org/" rel="noopener nofollow ugc">MLIR</a> The MLIR project is a novel approach to building reusable and extensible compiler infrastructure. MLIR aims to address software fragmentation, improve compilation for heterogeneous hardware, significantly reduce the cost of building domain specific compilers, and aid in connecting existing compilers together.</p>
<p><a href="https://github.com/llvm/torch-mlir" rel="noopener nofollow ugc">Torch-MLIR</a> Multiple Vendors use MLIR as the middle layer mapping from platform frameworks like PyTorch, JAX, TensorFlow onto MLIR and then progressively lowering down to their target hardware. We have seen half a dozen custom lowerings from PyTorch to MLIR. Having canonical lowerings from the PyTorch ecosystem to the MLIR ecosystem would provide much needed relief to hardware vendors to focus on their unique value rather than implementing another PyTorch frontend for MLIR. It would be similar to current hardware vendors adding LLVM target support instead of each one also implementing the Clang/C++ frontend.</p>
<p>All the roads from PyTorch to Torch MLIR Dialect</p>
<p>We have a few paths to lower down to the Torch MLIR Dialect.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/b/db9065d6617c49e71530635a7d4d6309fd333dc4.png" data-download-href="https://discuss.pytorch.org/uploads/default/db9065d6617c49e71530635a7d4d6309fd333dc4" title="Torch-MLIR"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db9065d6617c49e71530635a7d4d6309fd333dc4_2_690x492.png" alt="Torch-MLIR" data-base62-sha1="vklW5pTKXWR5GblcmpUkLlsGaxe" width="690" height="492" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db9065d6617c49e71530635a7d4d6309fd333dc4_2_690x492.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db9065d6617c49e71530635a7d4d6309fd333dc4_2_1035x738.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/b/db9065d6617c49e71530635a7d4d6309fd333dc4_2_1380x984.png 2x" data-dominant-color="F4F4F4"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Torch-MLIR</span><span class="informations">1780×1271 236 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<ul>
<li>TorchScript<br>
This is the most tested path from PyTorch to Torch MLIR Dialect.</li>
<li>TorchFX<br>
This provides a path to go from TorchFX down to MLIR. This a functional prototype.</li>
<li>Lazy Tensor Core (Based on lazy_tensor_staging branch)<br>
This path provides the upcoming LTC path of capture. It is based on an unstable devel branch but is the closest way for you to adapt any existing torch_xla derivatives.</li>
<li>“ACAP” (Deprecated torch_xla based capture, mentioned here for completeness)</li>
</ul>
<p>The torch-mlir project includes a few examples of lowering down via each path from PyTorch to MLIR and using the “mlir-cpu-runner” to target a CPU backend. Obviously this is just a starting point and you can import this project into your larger MLIR projects to continue lowering to target GPUs and other Accelerators.</p>
<p>We are looking for specific feedback on two fronts:</p>
<p><strong>Vendors</strong>: If you use your custom lowerings to get to MLIR eventually, is there anything else you would like to see. Please open github issues in torch-mlir.</p>
<p><strong>Pytorch Dev community</strong>:<br>
<a href="https://github.com/pytorch/pytorch/pull/65880" rel="noopener nofollow ugc">Here</a> is a PR to integrate Torch-MLIR with an optional build flag into PyTorch. There are included samples to export a ResNet Model and we plan to add BERT and MASK-RCNN examples next.  Please let us know what you think.</p>
<p>Also, any feedback on the various paths down to MLIR (TS, FX, LTC-&gt;TS, LTC-&gt;MLIR, etc) would be great.</p>
<p>Anush<br>
(For the torch-mlir team, nod.ai team and the others who contributed to this effort.)</p>

stephenneuendorffer: <p>As a silicon vendor and contributor to the torch-mlir project, Xilinx is very excited to see this maturing.  Having a stable path from pytorch to MLIR that is capable of supporting a wide variety of use cases for inference and training is going to be very helpful!</p>

Stella_Laurenzo: <p>As a part of the MLIR ecosystem at Google and a contributor to torch-mlir, we’re really supportive of seeing this mature into a great connection point between the PyTorch ecosystem and the MLIR ecosystem broadly. It will help a lot to have this component, done well and aligned with the PyTorch roadmap for interfacing to the existing and new APIs for program extraction.</p>

sjarus: <p>We at Arm are really excited about this effort. We contributed the TOSA dialect to the MLIR ecosystem, and this offers an interface between the PyTorch and MLIR ecosystems. It will help us develop pathways from PyTorch content to MLIR based code generation targets.</p>

Shirish_Seetharam: <p>This would be very useful!</p>

Mahesh_Makhijani: <p>As a Life-Ready edge AI provider, GrAI Matter Labs is supportive of this effort.</p>

cormacmb: <p>At Intel we are also supportive of this effort.  As we work more on MLIR lowering, it’s really useful to see a path to better connections between the pytorch ecosystem and the broader MLIR ecosystem.</p>

powderluv: <p>FYI: We will be discussing torch-mlir at the LLVM/MLIR ODM on Oct 7th 9am PT. Details <a href="https://docs.google.com/document/d/1y_9f1AbfgcoVdJh4_aM6-BaSHvrHl8zuA5G4jv_94K8/edit#heading=h.cite1kolful9" rel="noopener nofollow ugc">here</a> . Please join us if you are interested and thank you all for your support.</p>

mostafa.hagog: <p>We at SiFive support this effort; having a unified lowering from PyTorch to MLIR allows us to optimize the code-gen on RISC-V in general and SiFive core IP in specific while enabling such an important framework.</p>

byronyi: <p>We at ByteDance support this effort to further improve PyTorch performance on GPU and other silicons using MLIR backed compiler optimizations.</p>

mehdi_amini: <p>If you missed the meeting this morning, here are the <a href="https://mlir.llvm.org/OpenMeetings/2021-10-07-The-Torch-MLIR-project.pdf" rel="noopener nofollow ugc">slides</a> ​and the <a href="https://youtu.be/QbNkex-gizs" rel="noopener nofollow ugc">recording</a>.<br>
This is a very exciting direction, I’m looking forward to it! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=10" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

111453: <p>That’s exciting, it would be useful. We had already do some works on Torch to MHLO at Alibaba, such as DISC.</p>

powderluv: <p>We have pushed a second RFC PR here: <a href="https://github.com/pytorch/pytorch/pull/66536" class="inline-onebox" rel="noopener nofollow ugc">Add simple `torch.mlir.export` function and demo notebook. by silvasean · Pull Request #66536 · pytorch/pytorch · GitHub</a> to integrate with PyTorch. It follows the same api format as torch.onnx.export and provides a torch.mlir.export.  If there are any suggestions / feedback on this approach from the various interested parties vs the earlier integration RFC  (<a href="https://github.com/pytorch/pytorch/pull/65880" class="inline-onebox" rel="noopener nofollow ugc">Add torch-mlir MLIR Exporter to PyTorch build by powderluv · Pull Request #65880 · pytorch/pytorch · GitHub</a>) at the C++ level please let us know.  The Python integration could potentially work across multiple PyTorch versions.</p>

CaoZhong: <p><img src="https://discuss.pytorch.org/images/emoji/apple/tada.png?v=10" title=":tada:" class="emoji" alt=":tada:"><img src="https://discuss.pytorch.org/images/emoji/apple/tada.png?v=10" title=":tada:" class="emoji" alt=":tada:">This is definitely a good trend. <img src="https://discuss.pytorch.org/images/emoji/apple/tada.png?v=10" title=":tada:" class="emoji" alt=":tada:"><img src="https://discuss.pytorch.org/images/emoji/apple/tada.png?v=10" title=":tada:" class="emoji" alt=":tada:"></p>
<p>However I’m thinking PyTorch frontend need to be reinforced for better usage of MLIR.</p>

yetanadur: <p>is there a way to roundtrip from torch-mlir back to TS ?</p>

mkal: <p>As a hardware designer, this is great news but I’m hoping quantization support is fully supported. I’m trying to convert a quantized ONNX model with onnx-mlir and quantizelinear etc layers are not supported at all. I’m hoping this is different in torch-mlir.</p>