Converting facebook/opt-13b to onnx
Author: kargarisaac
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.21.0.dev0
- Platform: Linux-5.4.0-72-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.13
- Huggingface_hub version: 0.8.1
- PyTorch version (GPU?): 1.12.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help?

@SaulLu @patrickvonplaten @Narsil @gante @LysandreJik 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I used the PR [here](https://github.com/huggingface/transformers/pull/17771) to convert facebook/opt-13b to onnx. You can see the discussion I had [here](https://github.com/huggingface/optimum/issues/202). Now, when I want to do the conversion, after fixing some problems, I face a problem whith no error and I'm not sure how should I solve it.

I got opt-13, finetuned it on a custom dataset and saved it locally. The model is saved in 3 parts: `pytorch_model-00001-of-00003.bin`, `pytorch_model-00002-of-00003.bin` , and `pytorch_model-00003-of-00003.bin`.

![image](https://user-images.githubusercontent.com/24753756/180838890-e90b1be6-b97e-4870-ab57-e2d5cee9b964.png)

Now I want to convert it to onnx using `python -m transformers.onnx --model=local-pt-checkpoint --feature=causal-lm` but use the local saved model (local-pt-checkpoint) and not facebook/opt-13b from huggingfacehub.

Here is the code in jupyter on a system with enough memory:
```
!pip install git+https://github.com/kargarisaac/transformers.git@opt_onnx_13b
!pip install onnxruntime | tail -1
!pip install onnx | tail -1
!pip install "optimum[onnxruntime]==1.3.0" -q
```

```
!python -m transformers.onnx --model=local-folder/ onnx/ --feature=causal-lm 
```
Now I get the following message and no onnx file is saved. How can I solve this? It doesn't work for the main facebook/opt-13b model too.

```
Using framework PyTorch: 1.12.0
Overriding 1 configuration item(s)
	- use_cache -> False
```


### Expected behavior

I expect to save the converted onnx model. The same command works for all other opt versions but not for opt-13b. I forked the mentioned PR and changed it a bit, `tokenizer = AutoTokenizer.from_pretrained(model_name, use_fast=False)` because it was complaining about fast tokenizer. You can see the repo [here](https://github.com/kargarisaac/transformers.git) on branch `opt_onnx_13b`.

Comments:
@mfuntowicz Maybe for visibility on ONNX export.

I think the problem was memory. Now it works but I get the same output about use_cache which I assume is just a warning. Anyway the model is now converted. I have another problem now. I quantize the model using:
```python
from onnxruntime.quantization.calibrate import CalibrationMethod
from onnxruntime.quantization import quantize_dynamic, QuantType, quantize_static

quantize_dynamic(
    "onnx2/model.onnx",
    "onnx-quantized2/model-int8.onnx",
    weight_type=QuantType.QUInt8,
    use_external_data_format=True
)
```

The model output is not good at all now:
```
This is an award winning short story titled The Drive. This story titled The A
 A story  A A  A A A
 A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A A
```

It should be sth like this:
```
This is an award winning short story titled The Drive. This story is written with descriptive language, described in detail. This is the first chapter of The Drive.\n\nIn this chapter I am a professional driver. I am driving a car from San Francisco to Los Angeles. I have a female passenger who is a famous photographer. She is taking a photo of me as I drive.\n\n#action, #driving, #fiction, #funny, #funny, #driving,'
```

What do you think is the problem? 

I cannot tell. Is the ONNX model correct before quantization ?

@Narsil yes the output before quantization is ok. Any other tool you suggest I can use? Is DeepSpeed an option here or I have to use it during training? I don't have access to data now. Just the mode.

The only thing that stands out right here is:

```
weight_type=QuantType.QUInt8,
```
It's not necessarily wrong, but I don't recall having using it.

If it's the quantization that's causing a problem the best is to open an issue directly at onnxruntime I guess.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.