Inference benchmarks of Torchdynamo + FX2TRT(now in Torch-TensorRT)  
Author: frank-wei
Pull request: yes

Hey HF folks:
I propose this PR as performance results preview for our SOTA inference engine combining torchdynamo+fx2trt. This PR is the based on the sound and great work in #17240 (cc @Chillee @jansel). Since the importance of torchdynamo is already emphasized in #17240, I will skip it and focus on the inference efforts we proposed by combining torchdynamo + fx2trt.  
# A short introduction about fx2trt
It is library tool developed by Meta team (cc @yinghai) to lower the FX graph to TensorRT on GPU to take advantage of its various optimization paths. The library itself was just merged into [Torch-TensorRT](https://github.com/pytorch/TensorRT) as one of its two lowering paths to TensorRT running on GPU.
 
# Results
This script primarily comes from a great effort from @anijain2305 and I added some inference implementation logic there.  
Some highlights about the results:
1. Compared with FX integration, it is important that torchdynamo could handle those corner cases where FX could not like control flow or set item operation. That is the main the reason we could extend our implementation to these 12 models. Actually, maybe more models could be run but I just follow the same model pool from #17204 
2. The experiments are run across different batch sizes from 1 to 4 and 8. The speedup is compared against the eager model. The trend is that speedup is pretty high on small batch size like 1 but get slower down when the batch size is increased. The reason behind it could be that torch eager mode is low efficiency on handling the small computation kernel while on large kernels, TRT and eager model would call similar kernel implementations. So their perf gap is reduced on large batch size.
3. TensorRT has accuracy degradation problem in float16 mode because of its optimization techniques. I changed the accuracy validation standard as [Cosine Similarity](https://pytorch.org/docs/stable/generated/torch.nn.CosineSimilarity.html) which is also another important evaluation standard on the accuracy against eager mode. For float32 mode, the absolute difference is used as standard.

Run on A100:
```
To run for fp32 mode:
$ python hf_dynamo.py --run-dynamo-fx2trt-fp32 --use-eval-mode
 

To run for fp16 mode:
$ python hf_dynamo.py --run-dynamo-fx2trt-fp16 --use-eval-mode
```
cc @stas00  

=== Final results for fp16 ===
**Batch size = 1**

| model                      | dtype         | is_accurate   | name               |   time (s) |   mem (GB) |   speedup |   mem_compression |
|:---------------------------|:--------------|:--------------|:-------------------|-----------:|-----------:|----------:|------------------:|
| BertForMaskedLM            | torch.float16 | True          | eager              |      0.010 |      0.458 |     1.000 |             1.000 |
| BertForMaskedLM            | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.003 |      0.240 |     2.930 |             1.906 |
| AlbertForMaskedLM          | torch.float16 | True          | eager              |      0.012 |      0.417 |     1.000 |             1.000 |
| AlbertForMaskedLM          | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.004 |      0.051 |     2.775 |             8.257 |
| GPT2LMHeadModel            | torch.float16 | True          | eager              |      0.013 |      0.630 |     1.000 |             1.000 |
| GPT2LMHeadModel            | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.006 |      0.316 |     2.314 |             1.991 |
| T5ForConditionalGeneration | torch.float16 | True          | eager              |      0.021 |      0.504 |     1.000 |             1.000 |
| T5ForConditionalGeneration | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.024 |      0.162 |     0.898 |             3.115 |
| DistilBertForMaskedLM      | torch.float16 | True          | eager              |      0.006 |      0.273 |     1.000 |             1.000 |
| DistilBertForMaskedLM      | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.003 |      0.158 |     2.119 |             1.722 |
| RobertaForMaskedLM         | torch.float16 | True          | eager              |      0.011 |      0.506 |     1.000 |             1.000 |
| RobertaForMaskedLM         | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.006 |      0.290 |     1.986 |             1.743 |
| GPT2LMHeadModel            | torch.float16 | True          | eager              |      0.006 |      0.446 |     1.000 |             1.000 |
| GPT2LMHeadModel            | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.003 |      0.220 |     1.909 |             2.026 |
| ElectraForMaskedLM         | torch.float16 | True          | eager              |      0.010 |      0.458 |     1.000 |             1.000 |
| ElectraForMaskedLM         | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.003 |      0.241 |     3.044 |             1.900 |
| ConvBertForMaskedLM        | torch.float16 | True          | eager              |      0.020 |      0.459 |     1.000 |             1.000 |
| ConvBertForMaskedLM        | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.021 |      0.231 |     0.944 |             1.985 |
| MobileBertForMaskedLM      | torch.float16 | True          | eager              |      0.040 |      0.297 |     1.000 |             1.000 |
| MobileBertForMaskedLM      | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.006 |      0.128 |     6.693 |             2.312 |
| CamembertForMaskedLM       | torch.float16 | True          | eager              |      0.011 |      0.460 |     1.000 |             1.000 |
| CamembertForMaskedLM       | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.005 |      0.241 |     2.075 |             1.906 |
| LayoutLMForMaskedLM        | torch.float16 | True          | eager              |      0.011 |      0.466 |     1.000 |             1.000 |
| LayoutLMForMaskedLM        | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.004 |      0.247 |     3.031 |             1.885 |
  
**Batch size = 4**  

| model                      | dtype         | is_accurate   | name               |   time (s) |   mem (GB) |   speedup |   mem_compression |
|:---------------------------|:--------------|:--------------|:-------------------|-----------:|-----------:|----------:|------------------:|
| BertForMaskedLM            | torch.float16 | True          | eager              |      0.012 |      1.185 |     1.000 |             1.000 |
| BertForMaskedLM            | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.008 |      0.326 |     1.445 |             3.631 |
| AlbertForMaskedLM          | torch.float16 | True          | eager              |      0.013 |      1.455 |     1.000 |             1.000 |
| AlbertForMaskedLM          | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.005 |      0.137 |     2.508 |            10.610 |
| GPT2LMHeadModel            | torch.float16 | True          | eager              |      0.014 |      1.826 |     1.000 |             1.000 |
| GPT2LMHeadModel            | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.008 |      0.515 |     1.705 |             3.549 |
| T5ForConditionalGeneration | torch.float16 | True          | eager              |      0.025 |      1.684 |     1.000 |             1.000 |
| T5ForConditionalGeneration | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.024 |      0.291 |     1.055 |             5.778 |
| DistilBertForMaskedLM      | torch.float16 | True          | eager              |      0.007 |      0.687 |     1.000 |             1.000 |
| DistilBertForMaskedLM      | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.006 |      0.253 |     1.227 |             2.721 |
| RobertaForMaskedLM         | torch.float16 | True          | eager              |      0.015 |      1.288 |     1.000 |             1.000 |
| RobertaForMaskedLM         | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.012 |      0.427 |     1.246 |             3.012 |
| GPT2LMHeadModel            | torch.float16 | True          | eager              |      0.008 |      1.120 |     1.000 |             1.000 |
| GPT2LMHeadModel            | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.006 |      0.394 |     1.430 |             2.842 |
| ElectraForMaskedLM         | torch.float16 | True          | eager              |      0.013 |      1.186 |     1.000 |             1.000 |
| ElectraForMaskedLM         | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.008 |      0.329 |     1.517 |             3.605 |
| ConvBertForMaskedLM        | torch.float16 | True          | eager              |      0.021 |      1.243 |     1.000 |             1.000 |
| ConvBertForMaskedLM        | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.020 |      0.321 |     1.077 |             3.876 |
| MobileBertForMaskedLM      | torch.float16 | True          | eager              |      0.043 |      0.893 |     1.000 |             1.000 |
| MobileBertForMaskedLM      | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.009 |      0.221 |     4.923 |             4.046 |
| CamembertForMaskedLM       | torch.float16 | True          | eager              |      0.014 |      1.189 |     1.000 |             1.000 |
| CamembertForMaskedLM       | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.011 |      0.331 |     1.302 |             3.597 |
| LayoutLMForMaskedLM        | torch.float16 | True          | eager              |      0.012 |      1.189 |     1.000 |             1.000 |
| LayoutLMForMaskedLM        | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.009 |      0.330 |     1.443 |             3.599 |  

**Batch size = 8**  

| model                      | dtype         | is_accurate   | name               |   time (s) |   mem (GB) |   speedup |   mem_compression |
|:---------------------------|:--------------|:--------------|:-------------------|-----------:|-----------:|----------:|------------------:|
| BertForMaskedLM            | torch.float16 | True          | eager              |      0.015 |      2.156 |     1.000 |             1.000 |
| BertForMaskedLM            | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.014 |      0.439 |     1.014 |             4.916 |
| AlbertForMaskedLM          | torch.float16 | True          | eager              |      0.017 |      2.840 |     1.000 |             1.000 |
| AlbertForMaskedLM          | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.009 |      0.250 |     1.892 |            11.361 |
| GPT2LMHeadModel            | torch.float16 | True          | eager              |      0.021 |      3.393 |     1.000 |             1.000 |
| GPT2LMHeadModel            | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.015 |      0.780 |     1.376 |             4.349 |
| T5ForConditionalGeneration | torch.float16 | True          | eager              |      0.025 |      3.243 |     1.000 |             1.000 |
| T5ForConditionalGeneration | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.027 |      0.465 |     0.929 |             6.977 |
| DistilBertForMaskedLM      | torch.float16 | True          | eager              |      0.009 |      1.237 |     1.000 |             1.000 |
| DistilBertForMaskedLM      | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.009 |      0.371 |     0.904 |             3.334 |
| RobertaForMaskedLM         | torch.float16 | True          | eager              |      0.018 |      2.335 |     1.000 |             1.000 |
| RobertaForMaskedLM         | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.019 |      0.618 |     0.958 |             3.781 |
| GPT2LMHeadModel            | torch.float16 | True          | eager              |      0.013 |      2.001 |     1.000 |             1.000 |
| GPT2LMHeadModel            | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.010 |      0.626 |     1.285 |             3.197 |
| ElectraForMaskedLM         | torch.float16 | True          | eager              |      0.015 |      2.156 |     1.000 |             1.000 |
| ElectraForMaskedLM         | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.014 |      0.444 |     1.023 |             4.857 |
| ConvBertForMaskedLM        | torch.float16 | True          | eager              |      0.022 |      2.273 |     1.000 |             1.000 |
| ConvBertForMaskedLM        | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.024 |      0.439 |     0.937 |             5.180 |
| MobileBertForMaskedLM      | torch.float16 | True          | eager              |      0.042 |      1.685 |     1.000 |             1.000 |
| MobileBertForMaskedLM      | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.013 |      0.343 |     3.265 |             4.909 |
| CamembertForMaskedLM       | torch.float16 | True          | eager              |      0.016 |      2.171 |     1.000 |             1.000 |
| CamembertForMaskedLM       | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.017 |      0.453 |     0.950 |             4.792 |
| LayoutLMForMaskedLM        | torch.float16 | True          | eager              |      0.015 |      2.163 |     1.000 |             1.000 |
| LayoutLMForMaskedLM        | torch.float16 | True          | dynamo_fx2trt_fp16 |      0.015 |      0.445 |     1.010 |             4.859 |  

  
=== Final results for fp32 ===  
**Batch size = 1**    
| model                      | dtype         | is_accurate   | name               |   time (s) |   mem (GB) |   speedup |   mem_compression |
|:---------------------------|:--------------|:--------------|:-------------------|-----------:|-----------:|----------:|------------------:|
| BertForMaskedLM            | torch.float32 | True          | eager              |      0.010 |      0.905 |     1.000 |             1.000 |
| BertForMaskedLM            | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.004 |      0.468 |     2.371 |             1.936 |
| AlbertForMaskedLM          | torch.float32 | True          | eager              |      0.011 |      0.839 |     1.000 |             1.000 |
| AlbertForMaskedLM          | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.004 |      0.100 |     2.563 |             8.401 |
| GPT2LMHeadModel            | torch.float32 | True          | eager              |      0.011 |      1.237 |     1.000 |             1.000 |
| GPT2LMHeadModel            | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.005 |      0.616 |     2.163 |             2.009 |
| T5ForConditionalGeneration | torch.float32 | True          | eager              |      0.015 |      0.648 |     1.000 |             1.000 |
| T5ForConditionalGeneration | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.008 |      0.318 |     1.879 |             2.038 |
| DistilBertForMaskedLM      | torch.float32 | True          | eager              |      0.006 |      0.534 |     1.000 |             1.000 |
| DistilBertForMaskedLM      | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.003 |      0.313 |     2.080 |             1.705 |
| RobertaForMaskedLM         | torch.float32 | True          | eager              |      0.011 |      0.998 |     1.000 |             1.000 |
| RobertaForMaskedLM         | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.006 |      0.563 |     1.707 |             1.773 |
| GPT2LMHeadModel            | torch.float32 | True          | eager              |      0.006 |      0.879 |     1.000 |             1.000 |
| GPT2LMHeadModel            | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.003 |      0.429 |     1.987 |             2.050 |
| ElectraForMaskedLM         | torch.float32 | True          | eager              |      0.010 |      0.902 |     1.000 |             1.000 |
| ElectraForMaskedLM         | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.004 |      0.471 |     2.478 |             1.914 |
| ConvBertForMaskedLM        | torch.float32 | True          | eager              |      0.019 |      0.926 |     1.000 |             1.000 |
| ConvBertForMaskedLM        | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.017 |      0.462 |     1.088 |             2.004 |
| MobileBertForMaskedLM      | torch.float32 | True          | eager              |      0.039 |      0.593 |     1.000 |             1.000 |
| MobileBertForMaskedLM      | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.007 |      0.257 |     5.388 |             2.305 |
| CamembertForMaskedLM       | torch.float32 | True          | eager              |      0.011 |      0.904 |     1.000 |             1.000 |
| CamembertForMaskedLM       | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.006 |      0.474 |     1.833 |             1.908 |
| LayoutLMForMaskedLM        | torch.float32 | True          | eager              |      0.011 |      0.918 |     1.000 |             1.000 |
| LayoutLMForMaskedLM        | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.005 |      0.483 |     2.391 |             1.900 |
  
**Batch size = 4** 
   
| model                      | dtype         | is_accurate   | name               |   time (s) |   mem (GB) |   speedup |   mem_compression |
|:---------------------------|:--------------|:--------------|:-------------------|-----------:|-----------:|----------:|------------------:|
| BertForMaskedLM            | torch.float32 | True          | eager              |      0.015 |      2.361 |     1.000 |             1.000 |
| BertForMaskedLM            | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.011 |      0.643 |     1.334 |             3.672 |
| AlbertForMaskedLM          | torch.float32 | True          | eager              |      0.015 |      2.906 |     1.000 |             1.000 |
| AlbertForMaskedLM          | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.009 |      0.271 |     1.740 |            10.734 |
| GPT2LMHeadModel            | torch.float32 | True          | eager              |      0.016 |      3.631 |     1.000 |             1.000 |
| GPT2LMHeadModel            | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.012 |      1.012 |     1.412 |             3.589 |
| T5ForConditionalGeneration | torch.float32 | True          | eager              |      0.017 |      1.982 |     1.000 |             1.000 |
| T5ForConditionalGeneration | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.011 |      0.579 |     1.472 |             3.424 |
| DistilBertForMaskedLM      | torch.float32 | True          | eager              |      0.007 |      1.364 |     1.000 |             1.000 |
| DistilBertForMaskedLM      | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.007 |      0.498 |     1.109 |             2.741 |
| RobertaForMaskedLM         | torch.float32 | True          | eager              |      0.014 |      2.568 |     1.000 |             1.000 |
| RobertaForMaskedLM         | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.014 |      0.850 |     1.028 |             3.022 |
| GPT2LMHeadModel            | torch.float32 | True          | eager              |      0.009 |      2.227 |     1.000 |             1.000 |
| GPT2LMHeadModel            | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.007 |      0.777 |     1.336 |             2.866 |
| ElectraForMaskedLM         | torch.float32 | True          | eager              |      0.013 |      2.360 |     1.000 |             1.000 |
| ElectraForMaskedLM         | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.011 |      0.648 |     1.182 |             3.640 |
| ConvBertForMaskedLM        | torch.float32 | True          | eager              |      0.020 |      2.475 |     1.000 |             1.000 |
| ConvBertForMaskedLM        | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.019 |      0.639 |     1.054 |             3.874 |
| MobileBertForMaskedLM      | torch.float32 | True          | eager              |      0.041 |      1.783 |     1.000 |             1.000 |
| MobileBertForMaskedLM      | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.011 |      0.441 |     3.610 |             4.047 |
| CamembertForMaskedLM       | torch.float32 | True          | eager              |      0.013 |      2.375 |     1.000 |             1.000 |
| CamembertForMaskedLM       | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.013 |      0.658 |     1.010 |             3.610 |
| LayoutLMForMaskedLM        | torch.float32 | True          | eager              |      0.013 |      2.373 |     1.000 |             1.000 |
| LayoutLMForMaskedLM        | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.011 |      0.655 |     1.182 |             3.620 |
  
**Batch size = 8** 
   
| model                      | dtype         | is_accurate   | name               |   time (s) |   mem (GB) |   speedup |   mem_compression |
|:---------------------------|:--------------|:--------------|:-------------------|-----------:|-----------:|----------:|------------------:|
| BertForMaskedLM            | torch.float32 | True          | eager              |      0.024 |      4.309 |     1.000 |             1.000 |
| BertForMaskedLM            | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.020 |      0.874 |     1.184 |             4.928 |
| AlbertForMaskedLM          | torch.float32 | True          | eager              |      0.029 |      5.679 |     1.000 |             1.000 |
| AlbertForMaskedLM          | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.016 |      0.500 |     1.787 |            11.367 |
| GPT2LMHeadModel            | torch.float32 | True          | eager              |      0.031 |      6.769 |     1.000 |             1.000 |
| GPT2LMHeadModel            | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.022 |      1.548 |     1.436 |             4.374 |
| T5ForConditionalGeneration | torch.float32 | True          | eager              |      0.020 |      3.725 |     1.000 |             1.000 |
| T5ForConditionalGeneration | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.019 |      0.924 |     1.018 |             4.033 |
| DistilBertForMaskedLM      | torch.float32 | True          | eager              |      0.014 |      2.470 |     1.000 |             1.000 |
| DistilBertForMaskedLM      | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.012 |      0.743 |     1.115 |             3.325 |
| RobertaForMaskedLM         | torch.float32 | True          | eager              |      0.026 |      4.670 |     1.000 |             1.000 |
| RobertaForMaskedLM         | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.024 |      1.236 |     1.070 |             3.779 |
| GPT2LMHeadModel            | torch.float32 | True          | eager              |      0.018 |      3.993 |     1.000 |             1.000 |
| GPT2LMHeadModel            | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.013 |      1.243 |     1.356 |             3.211 |
| ElectraForMaskedLM         | torch.float32 | True          | eager              |      0.024 |      4.315 |     1.000 |             1.000 |
| ElectraForMaskedLM         | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.020 |      0.892 |     1.210 |             4.837 |
| ConvBertForMaskedLM        | torch.float32 | True          | eager              |      0.030 |      4.525 |     1.000 |             1.000 |
| ConvBertForMaskedLM        | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.031 |      0.882 |     0.991 |             5.133 |
| MobileBertForMaskedLM      | torch.float32 | True          | eager              |      0.044 |      3.371 |     1.000 |             1.000 |
| MobileBertForMaskedLM      | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.019 |      0.685 |     2.338 |             4.921 |
| CamembertForMaskedLM       | torch.float32 | True          | eager              |      0.024 |      4.338 |     1.000 |             1.000 |
| CamembertForMaskedLM       | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.022 |      0.903 |     1.098 |             4.802 |
| LayoutLMForMaskedLM        | torch.float32 | True          | eager              |      0.024 |      4.322 |     1.000 |             1.000 |
| LayoutLMForMaskedLM        | torch.float32 | True          | dynamo_fx2trt_fp32 |      0.020 |      0.888 |     1.185 |             4.869 |  


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_17724). All of your documentation changes will be reflected on that endpoint.

This looks excellent, @frank-wei, thank you for sharing the extensive benchmarks.

So the next step is to document how the users can deploy the proposed solution so that they don't need to try to fish it out from the benchmark code. Does it make sense?

wrt the benchmark script in this PR, I'm not sure where it'd best belong. Perhaps somewhere in your repo as I'm sure it's going to evolve and then we could link to it from our documentation? how does that sound?

also cc: @sgugger 

This should just be a simple addition to the existing TorchDynamo integration with NVFuser (cc: @anijain2305).

> This should just be a simple addition to the existing TorchDynamo integration with NVFuser (cc: @anijain2305).

If we do that, as I proposed originally looking into the 8 ball, it should go under the same cmd arg and have a new value - as we aren't going to add a new cmd arg for each variation. 

It'd like let's discuss the proposed integration API modifications before implementing those, to save everybody's time.

As I suggested probably the best future-proofing is to have the value comprised of possible multiple "keys" key1:key2:...:keyn - so that multiple combos could be supported down the road.

@stas00  and @Chillee  any context about the design of the integration API? Does the one API could work both for inference (fx2trt) and training (AOT)?

It's the one that was added recently to integrate torchdynamo with the nvfuser backend:
https://github.com/huggingface/transformers/blob/3981ee8650042e89d9c430ec34def2d58a2a12f7/src/transformers/training_args.py#L467-L469

you can see the PR here: https://github.com/huggingface/transformers/pull/17308

Since the fx2trt needs some preprocessing time to trace the model and create TRT model engine, it is not suitable for inference in training process. 
Does hf has pure inference scenarios where fx2trt could be leveraged?

> Does hf has pure inference scenarios where fx2trt could be leveraged?

Of course it has. Everything else besides training is inference

https://github.com/huggingface/transformers/blob/3981ee8650042e89d9c430ec34def2d58a2a12f7/src/transformers/training_args.py#L123-L127

https://github.com/huggingface/transformers/blob/3981ee8650042e89d9c430ec34def2d58a2a12f7/src/transformers/training_args.py#L267-L272


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@frank-wei have you tried `PEGASUS` ? It seemed to be not supported yet. I filled an issue with details: https://github.com/pytorch/torchdynamo/issues/777

> @frank-wei have you tried `PEGASUS` ? It seemed to be not supported yet. I filled an issue with details: [pytorch/torchdynamo#777](https://github.com/pytorch/torchdynamo/issues/777)

@philschmid , I did not try `PEGASUS` before. 
The problem in your case is that torch_tensorrt fx path nightly version is out of sync with pytorch nightly. I am updating it today.