Tracker - Failing models in the torch.compile dashboard
Labels: triaged, oncall: pt2

# Torchbench models


- [ ] @bdhirsh - Inductor(default) fails for `hf_Longformer` - Repro at https://github.com/pytorch/pytorch/issues/100067
Cmd - `python benchmarks/dynamo/torchbench.py --accuracy --training --amp --backend inductor --disable-cudagraphs --device cuda --total-partitions 3 --partition-id 1 --only=hf_Longformer`
Possible fix - https://github.com/pytorch/pytorch/pull/100115


- [x] @wconstab   - https://github.com/pytorch/pytorch/issues/103385
- [x] @williamwen42 - `hf_T5_base` is failing

- [x] @yanboliang - detectron2_maskrcnn - They dont show up in dashboard because they are skipped - https://github.com/pytorch/pytorch/issues/99665

# TIMM models

- [x] @eellison - Inductor (w/ cudagraphs) OOM for `cait_m36_384`
# Huggingface models

- [ ] (**Up for Grabs**)  -  Inductor (default) accuracy failure with `AlbertForQuestionAnswering` - Can't repro on AWS machine
Next steps - Repro this on GCP machine, check the offending tensors, increase tolerance if needed.

- [x] @eellison - Inductor (w/ cudagraphs) OOM for `DebertaV2ForQuestionAnswering` 



## Dynamic shapes (NOT POPULATED YET)


- [ ] @ezyang  - Inductor (dynamic) w/eval fails for `hf_BigBird` 
Cmd - `python benchmarks/dynamo/torchbench.py --accuracy --inference --amp --backend inductor --dynamic-shapes --dynamic-batch-only --disable-cudagraphs --device cuda --only=hf_BigBird`
~~~
2023-04-24T18:32:37.8937866Z     expr = pexpr(V.graph.sizevars.simplify(self.shape))
2023-04-24T18:32:37.8938395Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/sympy/printing/printer.py", line 292, in doprint
2023-04-24T18:32:37.8938757Z     return self._str(self._print(expr))
2023-04-24T18:32:37.8939235Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/sympy/printing/printer.py", line 331, in _print
2023-04-24T18:32:37.8939711Z     return printmethod(expr, **kwargs)
2023-04-24T18:32:37.8940289Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/_inductor/codegen/common.py", line 191, in _print_Pow
2023-04-24T18:32:37.8940643Z     assert exp.is_integer
2023-04-24T18:32:37.8941042Z torch._dynamo.exc.BackendCompilerFailed: backend='inductor' raised:
2023-04-24T18:32:37.8941357Z AssertionError: 
~~~

- [ ] @bdhirsh - Inductor(default) w/ training fails 
Cmd - `python benchmarks/dynamo/torchbench.py --accuracy --training --amp --backend inductor --disable-cudagraphs --device cuda --total-partitions 3 --partition-id 1 --only=hf_BigBird`
Error
~~~
2023-04-24T19:31:30.2966005Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 567, in bigbird_block_sparse_attention
2023-04-24T19:31:30.2966675Z     np.random.seed(seed)
2023-04-24T19:31:30.2967733Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 569, in <resume in bigbird_block_sparse_attention>
2023-04-24T19:31:30.2968585Z     rand_attn = [
2023-04-24T19:31:30.2969590Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 591, in <resume in bigbird_block_sparse_attention>
2023-04-24T19:31:30.2970293Z     rand_attn = np.stack(rand_attn, axis=0)
2023-04-24T19:31:30.2971020Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 592, in <resume in bigbird_block_sparse_attention>
2023-04-24T19:31:30.2971769Z     rand_attn = torch.tensor(rand_attn, device=query_layer.device, dtype=torch.long)
2023-04-24T19:31:30.2972585Z   File "/opt/conda/envs/py_3.10/lib/python3.10/site-packages/transformers/models/big_bird/modeling_big_bird.py", line 629, in <resume in bigbird_block_sparse_attention>
2023-04-24T19:31:30.2972999Z     first_context_layer.unsqueeze_(2)
2023-04-24T19:31:30.2974281Z RuntimeError: Output 0 of CompiledFunctionBackward is a view and is being modified inplace. This view was created inside a custom Function (or because an input was returned as-is) and the autograd logic to handle view+inplace would override the custom backward associated with the custom Function, leading to incorrect gradients. This behavior is forbidden. You can fix this by cloning the output of the custom Function.
~~~


- [ ] @anijain2305 + @ezyang - Inductor (w/ dynamic) fails for `convit_base` for sympy error

~~~
2023-04-24T18:14:43.2692442Z cuda train convit_base                         WARNING:common:fp64 golden ref were not generated for convit_base. Setting accuracy check to cosine
2023-04-24T18:14:58.0197194Z [2023-04-24 18:14:58,017] torch.fx.experimental.symbolic_shapes: [WARNING] 13.0: RecursionError in sympy.solve(floor(s0**0.5) - 14, s0)
2023-04-24T18:15:01.5186560Z ERROR:common:backend='inductor' raised:
2023-04-24T18:15:01.5187148Z CppCompileError: C++ compile error
2023-04-24T18:15:01.5187414Z 
2023-04-24T18:15:01.5187542Z Command:
2023-04-24T18:15:01.5190876Z g++ /tmp/torchinductor_jenkins/za/czarprmgsfhybui3toxkkq3vz6vil3qetzl7exfphbvz2kjjt4tr.cpp -shared -fPIC -Wall -std=c++17 -Wno-unused-variable -I/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/include -I/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -I/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/include/TH -I/opt/conda/envs/py_3.10/lib/python3.10/site-packages/torch/include/THC -I/opt/conda/envs/py_3.10/include/python3.10 -lgomp -O3 -ffast-math -fno-finite-math-only -march=native -fopenmp -D C10_USING_CUSTOM_GENERATED_MACROS -o /tmp/torchinductor_jenkins/za/czarprmgsfhybui3toxkkq3vz6vil3qetzl7exfphbvz2kjjt4tr.so
2023-04-24T18:15:01.5192575Z 
2023-04-24T18:15:01.5192698Z Output:
2023-04-24T18:15:01.5193959Z /tmp/torchinductor_jenkins/za/czarprmgsfhybui3toxkkq3vz6vil3qetzl7exfphbvz2kjjt4tr.cpp: In function â€˜void kernel(float*, long int*, long int*, long int)â€™:
2023-04-24T18:15:01.5195123Z /tmp/torchinductor_jenkins/za/czarprmgsfhybui3toxkkq3vz6vil3qetzl7exfphbvz2kjjt4tr.cpp:42:72: error: invalid operands of types â€˜long intâ€™ and â€˜doubleâ€™ to binary â€˜operator%â€™
2023-04-24T18:15:01.5196985Z                      auto tmp0 = out_ptr1[static_cast<long>((((i0 / 1L) % (std::floor(std::sqrt(ks0))))*(std::floor(std::sqrt(ks0)))) + ((i1 / 1L) % (std::floor(std::sqrt(ks0)))))];
2023-04-24T18:15:01.5197557Z                                                               ~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
2023-04-24T18:15:01.5198643Z /tmp/torchinductor_jenkins/za/czarprmgsfhybui3toxkkq3vz6vil3qetzl7exfphbvz2kjjt4tr.cpp:42:147: error: invalid operands of types â€˜long intâ€™ and â€˜doubleâ€™ to binary â€˜operator%â€™
2023-04-24T18:15:01.5202118Z                      auto tmp0 = out_ptr1[static_cast<long>((((i0 / 1L) % (std::floor(std::sqrt(ks0))))*(std::floor(std::sqrt(ks0)))) + ((i1 / 1L) % (std::floor(std::sqrt(ks0)))))];
~~~
Next steps
* @anijain2305 - Use this opportunity to test minifier with dynamic shapes
* @ezyang to fix/assign the owner to fix the issue



Completed
-------

- [x] Inductor (default) accuracy flakiness for `sebotnet33ts_256` - Fixed by @anijain2305 https://github.com/pytorch/pytorch/pull/99851

- [x] Inductor (default) w/eval fails for `hf_BigBird` - Fixed by @anijain2305

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh