Repository: pytorch/torchdynamo

Issue:
Assertions Found in HuggingFace's T5 Model
I am seeing a stack trace error in TorchDynamo that looks like it might be from this code in the T5 model.  I will note that I specifically edited the model to remove the Cuda Extension for RMSNorm that also produced an error in an attempt to better isolate the issue.  The Cuda Extension added an error as well that is not shown here.

File "/opt/conda/lib/python3.8/site-packages/transformers/modeling_utils.py", line 629, in invert_attention_mask

```
    def invert_attention_mask(self, encoder_attention_mask: Tensor) -> Tensor:
        """
        Invert an attention mask (e.g., switches 0. and 1.).

        Args:
            encoder_attention_mask (`torch.Tensor`): An attention mask.

        Returns:
            `torch.Tensor`: The inverted attention mask.
        """
        if encoder_attention_mask.dim() == 3:
            encoder_extended_attention_mask = encoder_attention_mask[:, None, :, :]
        if encoder_attention_mask.dim() == 2:
            encoder_extended_attention_mask = encoder_attention_mask[:, None, None, :]
        # T5 has a mask that can compare sequence ids, we can simulate this here with this transposition
        # Cf. https://github.com/tensorflow/mesh/blob/8d2465e9bc93129b913b5ccc6a59aa97abd96ec6/mesh_tensorflow
        # /transformer/transformer_layers.py#L270
        # encoder_extended_attention_mask = (encoder_extended_attention_mask ==
        # encoder_extended_attention_mask.transpose(-1, -2))
        encoder_extended_attention_mask = encoder_extended_attention_mask.to(dtype=self.dtype)  # fp16 compatibility
        encoder_extended_attention_mask = (1.0 - encoder_extended_attention_mask) * torch.finfo(self.dtype).min

        return encoder_extended_attention_mask
```

Here is the entire log output.  I am not sure what is helpful and/or relevant:

```
ERROR FROM offset=104 filename /opt/conda/lib/python3.8/site-packages/transformers/modeling_utils.py 649 AssertionError
========== TorchDynamo Stack Trace ==========
Traceback (most recent call last):
  File "/opt/conda/lib/python3.8/site-packages/torchdynamo/convert_frame.py", line 295, in _convert_frame_assert
    code = transform_code_object(frame.f_code, transform)
  File "/opt/conda/lib/python3.8/site-packages/torchdynamo/bytecode_transformation.py", line 338, in transform_code_object
    transformations(instructions, code_options)
  File "/opt/conda/lib/python3.8/site-packages/torchdynamo/convert_frame.py", line 271, in transform
    tracer.run()
  File "/opt/conda/lib/python3.8/site-packages/torchdynamo/symbolic_convert.py", line 307, in run
    and self.step()
  File "/opt/conda/lib/python3.8/site-packages/torchdynamo/symbolic_convert.py", line 285, in step
    getattr(self, inst.opname)(inst)
  File "/opt/conda/lib/python3.8/site-packages/torchdynamo/symbolic_convert.py", line 146, in wrapper
    return inner_fn(self, inst) 
  File "/opt/conda/lib/python3.8/site-packages/torchdynamo/symbolic_convert.py", line 621, in CALL_FUNCTION
    self.call_function(fn, args, {})
  File "/opt/conda/lib/python3.8/site-packages/torchdynamo/symbolic_convert.py", line 221, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "/opt/conda/lib/python3.8/site-packages/torchdynamo/variables/torch.py", line 193, in call_function
    tensor_variable = TensorVariable.create(
  File "/opt/conda/lib/python3.8/site-packages/torchdynamo/variables/tensor.py", line 179, in create
    assert (
AssertionError: torch.* op returned non-Tensor finfo call_function <class 'torch.finfo'>
========== Exception (above) while processing ==========
  File "huggingface_t5.py", line 43, in <module>
    final_results += runner.run(sys.argv, 'T5ForConditionalGeneration_t5-large_[seqs=8,src_seql=512,tgt_seql=128]', T5ForConditionalGeneration(config), optim_func, t5_input_func, None)
  File "/workspace/simple_dl_models/execution/runner.py", line 77, in run
    result_records.append(execution_loop.execute(args, name, model_name, model, optim_func, input_func, grad_func, eager_record))
  File "/workspace/simple_dl_models/execution/execution_loop.py", line 100, in execute
    loss = model(*batch)
  File "/opt/pytorch/pytorch/torch/nn/modules/module.py", line 1147, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 1638, in forward
    decoder_outputs = self.decoder(
  File "/opt/pytorch/pytorch/torch/nn/modules/module.py", line 1147, in _call_impl
    return forward_call(*input, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py", line 968, in forward
    encoder_extended_attention_mask = self.invert_attention_mask(encoder_attention_mask)
  File "/opt/conda/lib/python3.8/site-packages/transformers/modeling_utils.py", line 629, in invert_attention_mask
    def invert_attention_mask(self, encoder_attention_mask: Tensor) -> Tensor:
========== End debug info ==========
torchdynamo hit recompilation cache limit (64) for function '__getitem__' (/opt/pytorch/pytorch/torch/nn/modules/container.py:194), due to the following guard failures: [['___check_obj_id(self, 140121547442400)'], ['___check_obj_id(self, 140121547442400)'], ['___check_obj_id(self, 140121547442400)'], ['___check_obj_id(self, 140121547442400)'], ['___check_obj_id(self, 140121547442400)'], ['___check_obj_id(self, 140121547442400)'], ['___check_obj_id(self, 140121547442400)'], ['___check_obj_id(self, 140121547442400)'], ['___check_obj_id(self, 140121547442400)'], ['___check_obj_id(self, 140121547442400)'], ['___check_obj_id(self, 140121547442400)'], ['___check_obj_id(self, 140121547442400)'], ['___check_obj_id(self, 140121547442400)'], ['___check_obj_id(self, 140121547442400)'], ['___check_obj_id(self, 140121547442400)'], ['___check_obj_id(self, 140121547442400)'], ['___check_obj_id(self, 140121547442400)'], ['___check_obj_id(self, 140121547442400)'], ['___check_obj_id(self, 140121547442400)'], ['___check_obj_id(self, 140121547442400)'], ['___check_obj_id(self, 140121547442400)'], ['___check_obj_id(self, 140121547442400)'], ['___check_obj_id(self, 140121547442400)'], ['___check_obj_id(self, 140121547442400)'], ['idx == 0'], ['___check_obj_id(self, 140121547442400)'], ['idx == 0'], ['___check_obj_id(self, 140121547442400)'], ['idx == 0'], ['___check_obj_id(self, 140121547442400)'], ['idx == 0'], ['___check_obj_id(self, 140121547442400)'], ['idx == 0'], ['___check_obj_id(self, 140121547442400)'], ['idx == 0'], ['___check_obj_id(self, 140121547442400)'], ['idx == 0'], ['___check_obj_id(self, 140121547442400)'], ['idx == 0'], ['___check_obj_id(self, 140121547442400)'], ['idx == 0'], ['___check_obj_id(self, 140121547442400)'], ['idx == 0'], ['___check_obj_id(self, 140121547442400)'], ['idx == 0'], ['___check_obj_id(self, 140121547442400)'], ['idx == 0'], ['___check_obj_id(self, 140121547442400)'], ['idx == 0'], ['___check_obj_id(self, 140121547442400)'], ['idx == 0'], ['___check_obj_id(self, 140121547442400)'], ['idx == 0'], ['___check_obj_id(self, 140121547442400)'], ['idx == 0'], ['___check_obj_id(self, 140121547442400)'], ['idx == 0'], ['___check_obj_id(self, 140121547442400)'], ['idx == 0'], ['___check_obj_id(self, 140121547442400)'], ['idx == 0'], ['___check_obj_id(self, 140121547442400)'], ['idx == 0'], ['___check_obj_id(self, 140121547442400)']]to diagnose recompilation issues, see https://github.com/pytorch/torchdynamo/blob/main/TROUBLESHOOTING.md.
torchdynamo hit recompilation cache limit (64) for function 'forward' (/opt/conda/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py:632), due to the following guard failures: [['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)'], ['___check_obj_id(self, 140121547442256)']]to diagnose recompilation issues, see https://github.com/pytorch/torchdynamo/blob/main/TROUBLESHOOTING.md.
torchdynamo hit recompilation cache limit (64) for function 'forward' (/opt/conda/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py:560), due to the following guard failures: [['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ["tensor 'position_bias' strides mismatch at index 0. expected 16, actual 4194304"], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)'], ['___check_obj_id(self, 140121548765216)']]to diagnose recompilation issues, see https://github.com/pytorch/torchdynamo/blob/main/TROUBLESHOOTING.md.
torchdynamo hit recompilation cache limit (64) for function 'forward' (/opt/conda/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py:437), due to the following guard failures: [['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ["tensor 'position_bias' strides mismatch at index 0. expected 16, actual 262144"], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)'], ['___check_obj_id(self, 140121504837584)']]to diagnose recompilation issues, see https://github.com/pytorch/torchdynamo/blob/main/TROUBLESHOOTING.md.
torchdynamo hit recompilation cache limit (64) for function 'project' (/opt/conda/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py:475), due to the following guard failures: [['___check_obj_id(proj_layer, 140121547466304)'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid'], ['___guarded_code.valid']]to diagnose recompilation issues, see https://github.com/pytorch/torchdynamo/blob/main/TROUBLESHOOTING.md.
```

To reproduce:

```
git clone https://github.com/kevinstephano/simple_dl_models.git
cd simple_dl_models
python huggingface_t5.py --torchdynamo
```


Discussion:
Looks like we don't handle `torch.finfo()`, should be an easy fix I'll submit a patch.

Resolving patch:
diff --git a/torchdynamo/variables/torch.py b/torchdynamo/variables/torch.py
--- a/torchdynamo/variables/torch.py
+++ b/torchdynamo/variables/torch.py
@@ -76,6 +76,8 @@ def as_python_constant(self):
     def can_constant_fold_through(self):
         if self.value in (
             torch.device,
+            torch.finfo,
+            torch.iinfo,
             torch.is_tensor,
             torch.is_floating_point,
             torch.overrides.is_tensor_like,


Test patch:
diff --git a/tests/test_functions.py b/tests/test_functions.py
--- a/tests/test_functions.py
+++ b/tests/test_functions.py
@@ -67,6 +67,11 @@ def test_constant4(a, b):
             return a - b
         return b - a
 
+    @make_test
+    def test_finfo(a, b):
+        if torch.iinfo(torch.int32).bits == 32:
+            return torch.finfo(a.dtype).min * b
+
     @make_test
     def test_globalfn(a, b):
         return sub(a, b)