[Inductor][amp] clip: expected scalar type BFloat16 but found Float
Labels: triaged

### 🐛 Describe the bug

As the title, for reproduce:


bash [inductor_single_test.sh](https://github.com/chuanqi129/inductor-tools/blob/yudong/aws_auto/scripts/modelbench/inductor_single_run.sh) multiple inference performance torchbench clip amp


Error message:
```
loading model: 0it [00:00, ?it/s]
loading model: 0it [00:01, ?it/s]cpu  eval  clip                               

ERROR:common:expected scalar type BFloat16 but found Float
Traceback (most recent call last):
  File "/workspace/pytorch/benchmarks/dynamo/common.py", line 1997, in check_accuracy
    correct_result = self.run_n_iterations(
  File "/workspace/pytorch/benchmarks/dynamo/common.py", line 1882, in run_n_iterations
    self.model_iter_fn(mod, inputs, collect_outputs=False)
  File "benchmarks/dynamo/torchbench.py", line 437, in forward_pass
    return mod(*inputs)
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torchmultimodal/models/clip/model.py", line 72, in forward
    embeddings_a = self.encoder_a(features_a)
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torchmultimodal/models/clip/image_encoder.py", line 109, in forward
    x = self.encoder(x)
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/workspace/pytorch/torch/nn/modules/transformer.py", line 361, in forward
    output = mod(output, src_mask=mask, is_causal=is_causal, src_key_padding_mask=src_key_padding_mask_for_layers)
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/workspace/pytorch/torch/nn/modules/transformer.py", line 678, in forward
    x = x + self._sa_block(self.norm1(x), src_mask, src_key_padding_mask, is_causal=is_causal)
  File "/workspace/pytorch/torch/nn/modules/transformer.py", line 689, in _sa_block
    x = self.self_attn(x, x, x,
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/workspace/pytorch/torch/nn/modules/activation.py", line 1182, in forward
    return torch._native_multi_head_attention(
RuntimeError: expected scalar type BFloat16 but found Float
eager_1st_run_fail

```

### Versions

torchbench : 770d5cf7
torch :  2.1.0a0+git4734e4d
torchvision :  0.16.0a0+58366ab
torchtext :  0.16.0a0+c11d758
torchaudio :  2.1.0a0+dc83b38
torchdata :  0.7.0a0+1d231d1
dynamo_benchmarks :  f228c8b