Guard Failures in T5 Model
Labels: triaged, bug, oncall: pt2, module: dynamo

In addition to the `torch.finfo` issue, that I verified was fixed, there remains several guard failures.  Are these unique instances or the hierarchy of the failure?

The last guard failure provided the most specific point in the code dealing with the shape of the input linear matmuls of the multihead attention block:

```
        def project(hidden_states, proj_layer, key_value_states, past_key_value):
            """projects hidden states correctly to key/query states"""
            if key_value_states is None:
                # self-attn
                # (batch_size, n_heads, seq_length, dim_per_head)
                hidden_states = shape(proj_layer(hidden_states))
            elif past_key_value is None:
                # cross-attn
                # (batch_size, n_heads, seq_length, dim_per_head)
                hidden_states = shape(proj_layer(key_value_states))

            if past_key_value is not None:
                if key_value_states is None:
                    # self-attn
                    # (batch_size, n_heads, key_length, dim_per_head)
                    hidden_states = torch.cat([past_key_value, hidden_states], dim=2)
                else:
                    # cross-attn
                    hidden_states = past_key_value
            return hidden_states
```

```
torchdynamo hit recompilation cache limit (64) for function '__getitem__' (/opt/pytorch/pytorch/torch/nn/modules/container.py:194), due to the following guard failures: [['___check_obj_id(self, 140428461148432)'], ['___check_obj_id(self, 140428461148432)'], ['___check_obj_id(self, 140428461148432)'], ['___check_obj_id(self, 140428461148432)'], ['___check_obj_id(self, 140428461148432)'], ['___check_obj_id(self, 140428461148432)'], ['___check_obj_id(self, 140428461148432)'], ['___check_obj_id(self, 140428461148432)'], ['___check_obj_id(self, 140428461148432)'], ['___check_obj_id(self, 140428461148432)'], ['___check_obj_id(self, 140428461148432)'], ['___check_obj_id(self, 140428461148432)'], ['___check_obj_id(self, 140428461148432)'], ['___check_obj_id(self, 140428461148432)'], ['___check_obj_id(self, 140428461148432)'], ['___check_obj_id(self, 140428461148432)'], ['___check_obj_id(self, 140428461148432)'], ['___check_obj_id(self, 140428461148432)'], ['___check_obj_id(self, 140428461148432)'], ['___check_obj_id(self, 140428461148432)'], ['___check_obj_id(self, 140428461148432)'], ['___check_obj_id(self, 140428461148432)'], ['___check_obj_id(self, 140428461148432)'], ['___check_obj_id(self, 140428461148432)'], ['idx == 0'], ['___check_obj_id(self, 140428461148432)'], ['idx == 0'], ['___check_obj_id(self, 140428461148432)'], ['idx == 0'], ['___check_obj_id(self, 140428461148432)'], ['idx == 0'], ['___check_obj_id(self, 140428461148432)'], ['idx == 0'], ['___check_obj_id(self, 140428461148432)'], ['idx == 0'], ['___check_obj_id(self, 140428461148432)'], ['idx == 0'], ['___check_obj_id(self, 140428461148432)'], ['idx == 0'], ['___check_obj_id(self, 140428461148432)'], ['idx == 0'], ['___check_obj_id(self, 140428461148432)'], ['idx == 0'], ['___check_obj_id(self, 140428461148432)'], ['idx == 0'], ['___check_obj_id(self, 140428461148432)'], ['idx == 0'], ['___check_obj_id(self, 140428461148432)'], ['idx == 0'], ['___check_obj_id(self, 140428461148432)'], ['idx == 0'], ['___check_obj_id(self, 140428461148432)'], ['idx == 0'], ['___check_obj_id(self, 140428461148432)'], ['idx == 0'], ['___check_obj_id(self, 140428461148432)'], ['idx == 0'], ['___check_obj_id(self, 140428461148432)'], ['idx == 0'], ['___check_obj_id(self, 140428461148432)'], ['idx == 0'], ['___check_obj_id(self, 140428461148432)'], ['idx == 0'], ['___check_obj_id(self, 140428461148432)']]to diagnose recompilation issues, see https://github.com/pytorch/torchdynamo/blob/main/TROUBLESHOOTING.md.
torchdynamo hit recompilation cache limit (64) for function 'forward' (/opt/conda/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py:632), due to the following guard failures: [['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)'], ['___check_obj_id(self, 140428461148288)']]to diagnose recompilation issues, see https://github.com/pytorch/torchdynamo/blob/main/TROUBLESHOOTING.md.
torchdynamo hit recompilation cache limit (64) for function 'forward' (/opt/conda/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py:560), due to the following guard failures: [['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ["tensor 'position_bias' strides mismatch at index 0. expected 16, actual 4194304"], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)'], ['___check_obj_id(self, 140428418168912)']]to diagnose recompilation issues, see https://github.com/pytorch/torchdynamo/blob/main/TROUBLESHOOTING.md.
torchdynamo hit recompilation cache limit (64) for function 'forward' (/opt/conda/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py:437), due to the following guard failures: [['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ["tensor 'position_bias' strides mismatch at index 0. expected 16, actual 262144"], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)'], ['___check_obj_id(self, 140428219191360)']]to diagnose recompilation issues, see https://github.com/pytorch/torchdynamo/blob/main/TROUBLESHOOTING.md.
torchdynamo hit recompilation cache limit (64) for function 'project' (/opt/conda/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py:475), due to the following guard failures: [['___check_obj_id(proj_layer, 140428461148960)'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid']]to diagnose recompilation issues, see https://github.com/pytorch/torchdynamo/blob/main/TROUBLESHOOTING.md.
```

How to reproduce (the script automatically installs HuggingFace so you need to do nothing to install dependencies):
```
git clone https://github.com/kevinstephano/simple_dl_models.git
cd simple_dl_models
python huggingface_t5.py [--nvprims_nvfuser|--inductor]
```

For small GPUs, you might need to dial down the number of sequences to fit.

cc @ezyang @soumith @msaroufim @wconstab @ngimel @bdhirsh @voznesenskym @yanboliang @penguinwu @anijain2305 @EikanWang @jgong5 @Guobing-Chen @XiaobingSuper @zhuhaozhe @blzheng @Xia-Weiwen @wenzhe-nrv @jiayisunx @desertfire