Cannot Convert CLIP to TensorRT
Author: junwang-wish
Pull request: no

### System Info

- `transformers` version: 4.26.1
- Platform: Linux-4.14.301-224.520.amzn2.x86_64-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.14.1
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@amyeroberts

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

This is my code for exporting CLIP image encoder ( openai/clip-vit-large-patch14-336 ) as ONNX
```
import os
from transformers import AutoConfig, AutoProcessor, CLIPModel, CLIPVisionModel
from transformers.modeling_outputs import BaseModelOutputWithPooling
from pathlib import Path
from transformers.onnx import export
from typing import Mapping, OrderedDict
from transformers.onnx import OnnxConfig, validate_model_outputs
from functools import partial
import torch

class CLIPImageEncoder(CLIPModel):
    def forward(self,
        pixel_values: torch.FloatTensor
    ):
        outputs = self.get_image_features(
            pixel_values=pixel_values
        )
        return BaseModelOutputWithPooling(
            pooler_output=outputs.reshape(-1, 768)
        )

class EncoderOnnxConfig(OnnxConfig):
    @property
    def inputs(self) -> Mapping[str, Mapping[int, str]]:
        return OrderedDict(
            [
                ("pixel_values", {0: "batch", 1: "num_channels", 2: "height", 3: "width"})
            ]
        )

    @property
    def outputs(self) -> Mapping[str, Mapping[int, str]]:
        return OrderedDict(
            [
                ("pooler_output", {0: "batch", 1: "dim"})
            ]
        )


config = AutoConfig.from_pretrained("openai/clip-vit-large-patch14-336")
onnx_config = EncoderOnnxConfig(config)
model = CLIPImageEncoder.from_pretrained("openai/clip-vit-large-patch14-336")
del model.text_model
del model.text_projection
processor = AutoProcessor.from_pretrained("openai/clip-vit-large-patch14-336")

onnx_path = Path("tmp_onnx/model.onnx")

onnx_inputs, onnx_outputs = export(processor.image_processor, model, onnx_config, onnx_config.default_onnx_opset, onnx_path)

validate_model_outputs(
    onnx_config, processor.image_processor, model, onnx_path, onnx_outputs, 1e-4
)

```
It successfully outputs `model.onnx`, I then try to convert it to TensorRT within `triton inference server` by adding the following to `config.pbtxt`
```
optimization {
  graph { level: 3 }
  execution_accelerators {
    gpu_execution_accelerator : [ {
      name : "tensorrt"
      parameters { key: "precision_mode" value: "FP16" }
      parameters { key: "max_workspace_size_bytes" value: "1073741824" }
    }]
  }
}
``` 
but it outputs the following error
```
2023-05-09 15:12:36.763934678 [E:onnxruntime:log, tensorrt_execution_provider.h:58 log] [2023-05-09 15:12:36   ERROR] 10: [optimizer.cpp::computeCosts::3728] Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[onnx::MatMul_3130 + (Unnamed Layer* 135) [Shuffle].../visual_projection/MatMul]}.)
Segmentation fault (core dumped)
```
Given the large number of users of CLIP, Huggingface already made ONNX conversion step really smooth, if the exported ONNX can also be easily converted to TensorRT, that'd add a lot of value. 

I wonder if the error is due to specific implementation of CLIP in Huggingface repo? Like use of one operator instead of another although the outcome is the same.

### Expected behavior

I follow the same ONNX conversion script for many other models such as MiniLM, T5, DistilBert, and the resulting ONNX can be easily converted to TensorRT inside Triton Inference Server. This is not the case for CLIP (ViT) model. Ideally, all ONNXs exported by Huggingface can be easily converted to TensorRT inside Triton Inference Server.

Comments:
When it comes to production and deployment, you should use TensorFlow. This repo already supports TFCLIPModel and Triton Inference Server supports Tensorflow as well. I was able to convert some TF models in this repo into TensorRT without any bugs (including CLIP), and the success rate is 100%. For CLIP model, I recommend you use TF or `torch_tensorrt` (https://github.com/pytorch/TensorRT) to convert the model rather than ONNX path.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.