[RFC]: Moving most torch.compile backends out of core by 12/1/23
Labels: triaged, topic: bc breaking, dependency issue, oncall: pt2, module: dynamo

## TL;DR

If you own a backend in https://github.com/pytorch/pytorch/tree/main/torch/_dynamo/backends and we don't hear back from you **we would like to delete it from core by 12/1/23** - if you strongly disagree please let us know ASAP and explain below why you feel the custom registration mechanism is not suitable

This will be BC breaking

I realize this date might be frustrating but I've given a heads up to all the POCs listed below since March of this year so without some due date I worry this won't be prioritized.

## Problem

When `torch.compile()` was first released it came with a `backend` parameter that made it really easy to try out different compilers. The problem was that 
1. Many backends were not tested in CI so it was unclear what their expected pass rate was on the HF, TIMM or torchbench suite
2. The backends may not be running on the right hardware relative to CI
3. Because those backends are not maintained by the pytorch core/compiler team we can't offer the same support model to fix bugs
4. The backends will fail by default pending some optional dependencies that we can't add as pytorch dependencies or pytorch CI so out of the box users still need to install extra things
5. There is a perception that backends in core are higher quality than backends out of core which may or may not be the case, we don't want to be in the business of blessing some backends at the expense of others


## Solution

So the solution we proposed was an out of core registration mechanism highlighted here https://pytorch.org/docs/main/torch.compiler_custom_backends.html#registering-custom-backends which meant that end users of pytorch could retain the convenience of `torch.compile(..., backend="...")` without having backend specific code living in core. In fact we believe that having the code out of core will create a superior backend since it will increase dev velocity and the testing infrastructure can and should mimic what your end users care about. For us a good proxy for what users care about has been a combination of huggingface, timm and torchbench models https://github.com/pytorch/pytorch/tree/main/benchmarks/dynamo

If registering a custom backend is too clunky then please provide feedback on this RFC and we can make it better, our goal should be to have little to no friction imposed on end users relative to the current in core backends. Remember that for an existing in core backend they will fail by default unless you install something.

If the goal was to promote a new backend, again there is no requirement that the code needs to live in `pytorch/pytorch` we in fact have a precedent for promoting research backends like Hidet that used an out of core registration mechanism 
 which we featured prominently on our blog and retweeted https://pytorch.org/blog/introducing-hidet/

So we want to really separate marketing vs technical reasons for code to be in core so the general principle for new dynamo/compile backends is if your code will be maintained by pytorch core/compiler engineers it should be in core and if it is not then the Linux foundation marketing team can help you out.

## Criteria to market a backend

Again we don't want to be in the business of blessing or endorsing backends, what matters most is what customers use and care about and not what I care about. So that end in order for us to market a backend there are some quality guidelines that will be helpful to follow so you can use this as a checklist before engaging with the Linux foundation

1. The backend has a clear list of maintainers
2. The backend has been maintained for at least 6 months
3. The pass rate is comparable to inductor, see https://hud.pytorch.org/benchmark/compilers
4. The perf speedups are comparable to inductor, see https://hud.pytorch.org/benchmark/compilers

You can leverage our benchmark scripts as a starting point to prove out 3,4 https://github.com/pytorch/pytorch/tree/main/benchmarks/dynamo and moving forward we'll use this criteria to for example include a backend here https://pytorch.org/docs/main/torch.compiler.html - Again this does not mean that your backend is pytorch blessed/approved/supported it's a just proxy for what we think is a good backend 

## Backend Tracker

We already have a precedent for not accepting new backends https://github.com/pytorch/pytorch/pull/104426 but the next step is making sure that all backends in core are maintained by the pytorch core/compiler team

These are the backends that we should move out of core

* [x] IPEX - this is registered out core  https://github.com/pytorch/pytorch/pull/104329 - owner @ashokei
* [x] NVfuser was already removed https://github.com/pytorch/pytorch/pull/105789 - owner @IvanYashchuk 
* [ ] ONNXRT - owner @abock 
* [ ] TensorRT - this is already a shim backend and can be removed in fact the out of core registration mechanism already exists https://github.com/pytorch/TensorRT/pull/2311 - owner @narendasan 
* [ ] TorchXLA - owner @JackCaoG 
* [ ] TVM - owner @shingjan

cc @ezyang @wconstab @bdhirsh @anijain2305 @voznesenskym @penguinwu @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @chenyang78 @aakhundov @kadeng @malfet @gottbrath @gchanan