Export+AOTInductor issue tracker
Labels: triaged, tracker

Updated (October 20)

- 14K github models (72%) (https://github.com/jansel/pytorch-jit-paritybench)
  - [ ] #111691
  - [ ] #111693
  - [ ] 159 errors like: RuntimeError: expected inputs vector size to be 2, but got 1 (example ./generated/test_lwpyr_CSP_pedestrian_detection_in_pytorch.py:L2Norm # pytest ./generated/test_lwpyr_CSP_pedestrian_detection_in_pytorch.py -k test_003)
  - [ ] 153 errors like: AssertionError: Failed to produce a graph during tracing. Tracing through 'f' must produce a single graph. (example ./generated/test_lucidrains_imagen_pytorch.py:UpsampleCombiner # pytest ./generated/test_lucidrains_imagen_pytorch.py -k test_007) (https://github.com/pytorch/pytorch/issues/111254)
  - [ ] 130 errors like: AssertionError: Mutating module attribute min_val during export. (example ./generated/test_NVlabs_GroupViT.py:Transformer # pytest ./generated/test_NVlabs_GroupViT.py -k test_004) (https://github.com/pytorch/pytorch/issues/105530)
  - [ ] 84 errors like: RuntimeError: AOTInductorModelContainerRun( model_container.get(), input_handles.data(), input_tensors.size(), output_handles.data(), output_handles.size(), stream_handle, proxy_executor_handle) API call failed at /home/binbao/.cache/torch_extensions/py310_cu121/aot_inductor/main.cpp, line 66 (example ./generated/test_sithu31296_semantic_segmentation.py:Downsample # pytest ./generated/test_sithu31296_semantic_segmentation.py -k test_015)
  - [ ] 81 errors like: AssertionError: original output # 2 is None, but only the following types are supported: (<class 'torch.Tensor'>, <class 'torch.SymInt'>, <class 'torch.SymFloat'>, <class 'torch.SymBool'>) (example ./generated/test_elbayadm_attn2d.py:GridGatedMAX # pytest ./generated/test_elbayadm_attn2d.py -k test_011) (https://github.com/pytorch/pytorch/issues/111250)
  - [ ] 75 errors like: Unsupported: setattr(UserDefinedObjectVariable) <function Module.__setattr__ at 0x7f61e53808b0> (example ./generated/test_XPixelGroup_BasicSR.py:SFTUpBlock # pytest ./generated/test_XPixelGroup_BasicSR.py -k test_029)
  - [ ] 68 errors like: Unsupported: call_function BuiltinVariable(setattr) [TensorVariable(), ConstantVariable(str), ConstantVariable(bool)] {} (example ./generated/test_cedias_Hierarchical_Sentiment.py:EmbedAttention # pytest ./generated/test_cedias_Hierarchical_Sentiment.py -k test_000)
  - [ ] 53 errors like: AssertionError: Dynamo attempts to add additional input during export: value=0.1767766952966369, source=NNModuleSource(base=AttrSource(base=NNModuleSource(base=AttrSource(base=LocalSource(local_name='self', cell_or_freevar=False), member='wscale')), member='scale')) (example ./generated/test_open_mmlab_mmgeneration.py:FullyConnectedLayer # pytest ./generated/test_open_mmlab_mmgeneration.py -k test_015) (https://github.com/pytorch/pytorch/issues/111255)
  - [ ] 52 errors like: RuntimeError: Found following user inputs located at [0] are mutated. This is currently banned in the aot_export workflow. (example ./generated/test_kuprel_min_dalle.py:AttentionBase # pytest ./generated/test_kuprel_min_dalle.py -k test_000)
  - [ ] 51 errors like: Unsupported: 'call_function Mish in skip_files Builtin Mish, filename is None' (example ./generated/test_ikostrikov_pytorch_a2c_ppo_acktr_gail.py:Categorical # pytest ./generated/test_ikostrikov_pytorch_a2c_ppo_acktr_gail.py -k test_001)
  - [ ] 46 errors like: UserError: Dynamic control flow is not supported at the moment. Please use functorch.experimental.control_flow.cond to explicitly capture the control flow. For more information about this error, see: https://pytorch.org/docs/main/generated/exportdb/index.html#cond-operands (example ./generated/test_maudzung_SFA3D.py:FocalLoss # pytest ./generated/test_maudzung_SFA3D.py -k test_001)
  - [ ] 43 errors like: AttributeError: 'int' object has no attribute 'device' (example ./generated/test_chengchunhsu_EveryPixelMatters.py:FCOSDiscriminator # pytest ./generated/test_chengchunhsu_EveryPixelMatters.py -k test_008)
  - [ ] 41 errors like: UserError: Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time.You will need to explicitly give hint to the compiler. Please take a look at constrain_as_value OR constrain_as_size APIs (example ./generated/test_mdv3101_CDeCNet.py:GHMC # pytest ./generated/test_mdv3101_CDeCNet.py -k test_008) (https://github.com/pytorch/pytorch/issues/111252)
  - [ ] 34 errors like: ValueError: tree_unflatten(values, spec): `values` has length 2 but the spec refers to a pytree that holds 1 items (*). (example ./generated/test_LoSealL_VideoSuperResolution.py:Srcnn # pytest ./generated/test_LoSealL_VideoSuperResolution.py -k test_035)
  - [ ] 34 errors like: AssertionError:  (example ./generated/test_ruotianluo_self_critical_pytorch.py:SublayerConnection # pytest ./generated/test_ruotianluo_self_critical_pytorch.py -k test_010)
  - [ ] 29 errors like: RuntimeError: Error building extension 'aot_inductor_v2': [1/2] c++ -MMD -MF main.o.d -DTORCH_EXTENSION_NAME=aot_inductor_v2 -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_gcc\" -DPYBIND11_STDLIB=\"_libstdcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1013\" -isystem /data/users/binbao/pytorch/torch/include -isystem /data/users/binbao/pytorch/torch/include/torch/csrc/api/include -isystem /data/users/binbao/pytorch/torch/include/TH -isystem /data/users/binbao/pytorch/torch/include/THC -isystem /usr/local/cuda-12.1/include -isystem /home/binbao/.conda/envs/pytorch-3.10/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=1 -fPIC -std=c++17 -c /home/binbao/.cache/torch_extensions/py310_cu121/aot_inductor/main.cpp -o main.o  (example ./generated/test_Vious_LBAM_Pytorch.py:GaussActivation # pytest ./generated/test_Vious_LBAM_Pytorch.py -k test_001)
  - [ ] #111711
  - [ ] 23 errors like: Unsupported: call_method NNModuleVariable() _get_backward_hooks [] {} (example ./generated/test_xheon_panoptic_reconstruction.py:InstanceNorm3d # pytest ./generated/test_xheon_panoptic_reconstruction.py -k test_011)

- Huggingface (passrate 91%)
  - [x] ~~#105242~~
  - [x] ~~“torch._dynamo.exc.InternalTorchDynamoError: 'str' object has no attribute 'size’”~~
  - [x] ~~XLNetLMHeadModel: #109655~~
  - [ ] #110096


- TIMM (passrate 98%)
  - [x] ~~crossvit_9_240: fail_accuracy~~
  - [x]  ~~deit_base_distilled_patch16_224: fail_accuracy~~
  - [x]  ~~dla102: fail_accuracy~~
  - [x]  ~~dpn107: fail_accuracy~~
  - [x]  ~~levit_128: fail_accuracy~~
  - [x] ~~volo_d1_224: fail_accuracy~~
  - [ ] #105530
  - [x] #108173

- Torchbench (passrate 78%) (repro cmd: `python benchmarks/dynamo/torchbench.py --bfloat16 --accuracy --inference --device cuda  --export-aot-inductor  --only  `)
  - [x] ~~Background_Matting: fail_accuracy~~
  - [x] ~~LearningToPaint: fail_accuracy~~
  - [x] ~~Super_SloMo: fail_accuracy~~
  - [x] ~~dcgan: fail_accuracy~~
  - [x] ~~nvidia_deeprecommender: fail_accuracy~~
  - [x] ~~pytorch_CycleGAN_and_pix2pix: fail_accuracy~~
  - [x] ~~pytorch_unet: fail_accuracy~~
  - [x] ~~#108697~~
  - [x] ~~#108699~~
- **Export/Dynamo problems:**
  - [ ] BERT_pytorch, llama, yolov3: Mutating module attribute during export, the same as https://github.com/pytorch/pytorch/issues/105530
  - [x] ~~#109894~~
  - [ ] #109884
  - [ ] #109885
  - [ ]  doctr_det_predictor: ERROR:common:call_method UserDefinedObjectVariable(morphologyEx) __call__ [TensorVariable(), ConstantVariable(int), NumpyNdarrayVariable()] {}
  - [ ] #108698
  - [ ] drq: AssertionError: traced result #0 (<class 'torchbenchmark.models.drq.drqutils.SquashedNormal'>) is not among graph-captured outputs (<class 'torch._subclasses.fake_tensor.FakeTensor'>, <class 'torch._subclasses.fake_tensor.FakeTensor'>, <class 'torch._subclasses.fake_tensor.FakeTensor'>, <class 'torch._subclasses.fake_tensor.FakeTensor'>, <class 'torch._subclasses.fake_tensor.FakeTensor'>, <class 'torch._subclasses.fake_tensor.FakeTensor'>, <class 'torch._subclasses.fake_tensor.FakeTensor'>, <class 'torch._subclasses.fake_tensor.FakeTensor'>, <class 'torch._subclasses.fake_tensor.FakeTensor'>, <class 'torch._subclasses.fake_tensor.FakeTensor'>) or original args (<class 'torch.Tensor'>)
  - [ ] hf_T5_generate: torch._dynamo.exc.Unsupported: call_function deepcopy in skip_files /opt/conda/envs/py_3.10/lib/python3.10/copy.py
  - [ ] #109895
  - [ ] pyhpc_isoneutral_mixing: Found following user inputs located at [16, 17, 18, 19, 20, 21, 22] are mutated. This is currently banned in the aot_export workflow.
  - [ ] soft_actor_critic: ERROR: mismatch between eager output (<class 'torchbenchmark.models.soft_actor_critic.nets.SquashedNormal'>) and traced output (list of tensors)
  - [ ] timm_efficientdet: torch._subclasses.fake_tensor.UnsupportedOperatorException: torchvision.nms.default
  - [ ] #105532
  - [ ] #105531
- **Inductor problems:**
  - [x] ~~clip, hf_BigBird: #109655~~
  - [ ] #111227
  - [ ] #110304
  - [ ] #110089
- **DATA DEPENDENT (will be skipped):**
  - [ ] (HF) AllenaiLongformerBase: Control flow on data-dependent [here](https://github.com/huggingface/transformers/blob/0a55d9f7376f72ad3ff296d4249840021b03bcc4/src/transformers/models/longformer/modeling_longformer.py#L601)
  - [ ] cm3leon_generate: ERROR:common:Dynamic control flow is not supported at the moment. Please use functorch.experimental.control_flow.cond to explicitly capture the control flow
  - [ ] detectron2_fcos_r_50_fpn: data-dependent error occurring on https://fburl.com/code/7x2ztpni (probably needs a constraint)
  - [ ] fastNLP_Bert: ERROR:common:Consider annotating your code using constrain_as_*(). It appears that you're trying to get a value out of symbolic int/float whose value is data-dependent (and thus we do not know the true value.)  The expression we were trying to evaluate is i0 + 2 > 512 (unhinted: i0 + 2 > 512).  Scroll up to see where each of these data-dependent accesses originally occurred.
  - [ ] hf_Longformer: torch._dynamo.exc.UserError: Consider annotating your code using constrain_as_*(). It appears that you're trying to get a value out of symbolic int/float whose value is data-dependent (and thus we do not know the true value.)  The expression we were trying to evaluate is Eq(i0, 1) (unhinted: Eq(i0, 1)).  Scroll up to see where each of these data-dependent accesses originally occurred.
  - [ ] hf_Reformer: torch._dynamo.exc.UserError: Dynamic control flow is not supported at the moment. Please use functorch.experimental.control_flow.cond to explicitly capture the control flow
  - [ ] opacus_cifar10: ERROR:common:Dynamic control flow is not supported at the moment. Please use functorch.experimental.control_flow.cond to explicitly capture the control flow
RROR:common:Tried to use data-dependent value in the subsequent computation. This can happen when we encounter unbounded dynamic value that is unknown during tracing time.You will need to explicitly give hint to the compiler. Please take a look at constrain_as_value OR constrain_as_size APIs
  - [ ] speech_transformer: ERROR:common:Dynamic slicing on data-dependent value is not supported