[Inductor] [cpu][amp]  Eager model failed to run for some torchbench models
Labels: triaged, module: cpu inductor

### 🐛 Describe the bug

For amp inference cpu benchmark, there are 9 torchbench models fails to included in calculations for passrate compare with fp32 test.

- detectron2_fasterrcnn_r_101_c4
- detectron2_fasterrcnn_r_101_dc5
- detectron2_fasterrcnn_r_101_fpn
- detectron2_fasterrcnn_r_50_c4
- detectron2_fasterrcnn_r_50_dc5
- detectron2_fasterrcnn_r_50_fpn
- detectron2_fcos_r_50_fpn
- doctr_det_predictor
- vision_maskrcnn


Error MSG：

for **detectron2** models:
```
loading model: 0it [00:00, ?it/s]
loading model: 0it [00:04, ?it/s]
WARNING:root:detectron2_fasterrcnn_r_101_c4 failed to load
Eager model failed to run
Traceback (most recent call last):
  File "/workspace/pytorch/benchmarks/dynamo/common.py", line 1858, in validate_model
    self.model_iter_fn(model, example_inputs)
  File "benchmarks/dynamo/torchbench.py", line 474, in forward_pass
    return mod(*inputs)
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1519, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1528, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 150, in forward
    return self.inference(batched_inputs)
  File "/opt/conda/lib/python3.8/site-packages/detectron2/modeling/meta_arch/rcnn.py", line 208, in inference
    proposals, _ = self.proposal_generator(images, features, None)
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1519, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1528, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/detectron2/modeling/proposal_generator/rpn.py", line 477, in forward
    proposals = self.predict_proposals(
  File "/opt/conda/lib/python3.8/site-packages/detectron2/modeling/proposal_generator/rpn.py", line 503, in predict_proposals
    return find_top_rpn_proposals(
  File "/opt/conda/lib/python3.8/site-packages/detectron2/modeling/proposal_generator/proposal_utils.py", line 121, in find_top_rpn_proposals
    keep = batched_nms(boxes.tensor, scores_per_img, lvl, nms_thresh)
  File "/opt/conda/lib/python3.8/site-packages/detectron2/layers/nms.py", line 20, in batched_nms
    return box_ops.batched_nms(boxes.float(), scores, idxs, iou_threshold)
  File "/opt/conda/lib/python3.8/site-packages/torchvision/ops/boxes.py", line 73, in batched_nms
    return _batched_nms_vanilla(boxes, scores, idxs, iou_threshold)
  File "/workspace/pytorch/torch/jit/_trace.py", line 1235, in wrapper
    return fn(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torchvision/ops/boxes.py", line 109, in _batched_nms_vanilla
    curr_keep_indices = nms(boxes[curr_indices], scores[curr_indices], iou_threshold)
  File "/opt/conda/lib/python3.8/site-packages/torchvision/ops/boxes.py", line 41, in nms
    return torch.ops.torchvision.nms(boxes, scores, iou_threshold)
  File "/workspace/pytorch/torch/_ops.py", line 760, in __call__
    return self._op(*args, **kwargs or {})
RuntimeError: dets should have the same type as scores

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/pytorch/benchmarks/dynamo/common.py", line 3432, in run
    ) = runner.load_model(
  File "benchmarks/dynamo/torchbench.py", line 410, in load_model
    self.validate_model(model, example_inputs)
  File "/workspace/pytorch/benchmarks/dynamo/common.py", line 1860, in validate_model
    raise NotImplementedError("Eager model failed to run") from e
NotImplementedError: Eager model failed to run
```
for **doctr_det_predictor**:
```
loading model: 0it [00:02, ?it/s]
WARNING:root:doctr_det_predictor failed to load
Eager model failed to run
Traceback (most recent call last):
  File "/workspace/pytorch/benchmarks/dynamo/common.py", line 1858, in validate_model
    self.model_iter_fn(model, example_inputs)
  File "benchmarks/dynamo/torchbench.py", line 474, in forward_pass
    return mod(*inputs)
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1519, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1528, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/doctr/models/detection/differentiable_binarization/pytorch.py", line 211, in forward
    preds[0] for preds in self.postprocessor(prob_map.detach().cpu().permute((0, 2, 3, 1)).numpy())
TypeError: Got unsupported ScalarType BFloat16

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/pytorch/benchmarks/dynamo/common.py", line 3432, in run
    ) = runner.load_model(
  File "benchmarks/dynamo/torchbench.py", line 410, in load_model
    self.validate_model(model, example_inputs)
  File "/workspace/pytorch/benchmarks/dynamo/common.py", line 1860, in validate_model
    raise NotImplementedError("Eager model failed to run") from e
NotImplementedError: Eager model failed to run

```
For **vision_maskrcnn**:

```
WARNING:root:vision_maskrcnn failed to load
Eager model failed to run
Traceback (most recent call last):
  File "/workspace/pytorch/benchmarks/dynamo/common.py", line 1858, in validate_model
    self.model_iter_fn(model, example_inputs)
  File "benchmarks/dynamo/torchbench.py", line 474, in forward_pass
    return mod(*inputs)
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1519, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1528, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torchvision/models/detection/generalized_rcnn.py", line 104, in forward
    proposals, proposal_losses = self.rpn(images, features, targets)
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1519, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/workspace/pytorch/torch/nn/modules/module.py", line 1528, in _call_impl
    return forward_call(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torchvision/models/detection/rpn.py", line 372, in forward
    boxes, scores = self.filter_proposals(proposals, objectness, images.image_sizes, num_anchors_per_level)
  File "/opt/conda/lib/python3.8/site-packages/torchvision/models/detection/rpn.py", line 288, in filter_proposals
    keep = box_ops.batched_nms(boxes, scores, lvl, self.nms_thresh)
  File "/opt/conda/lib/python3.8/site-packages/torchvision/ops/boxes.py", line 73, in batched_nms
    return _batched_nms_vanilla(boxes, scores, idxs, iou_threshold)
  File "/workspace/pytorch/torch/jit/_trace.py", line 1235, in wrapper
    return fn(*args, **kwargs)
  File "/opt/conda/lib/python3.8/site-packages/torchvision/ops/boxes.py", line 109, in _batched_nms_vanilla
    curr_keep_indices = nms(boxes[curr_indices], scores[curr_indices], iou_threshold)
  File "/opt/conda/lib/python3.8/site-packages/torchvision/ops/boxes.py", line 41, in nms
    return torch.ops.torchvision.nms(boxes, scores, iou_threshold)
  File "/workspace/pytorch/torch/_ops.py", line 760, in __call__
    return self._op(*args, **kwargs or {})
RuntimeError: "nms_kernel" not implemented for 'BFloat16'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/workspace/pytorch/benchmarks/dynamo/common.py", line 3432, in run
    ) = runner.load_model(
  File "benchmarks/dynamo/torchbench.py", line 410, in load_model
    self.validate_model(model, example_inputs)
  File "/workspace/pytorch/benchmarks/dynamo/common.py", line 1860, in validate_model
    raise NotImplementedError("Eager model failed to run") from e
NotImplementedError: Eager model failed to run

```

<p>Repro</p>
<a href=https://github.com/chuanqi129/inductor-tools/blob/yudong/aws_auto/scripts/modelbench/inductor_single_run.sh>
  inductor_single_run.sh </a>
<code>
bash inductor_single_run.sh multiple inference performance torchbench model amp first static default 0
</code>


### Versions

torchbench : ffbbebb9
torch :  2.2.0a0+git20217d1
torchvision :  0.16.0a0+8636bf3
torchtext :  0.16.0a0+142d029
torchaudio :  2.2.0a0+475b6ae
torchdata :  0.7.0a0+eb9bf61
dynamo_benchmarks : 0200b11