Is it possible to use T5EncoderModel with torch_dtype=torch.float8_e4m3fn, or is this a bug?
Author: camenduru
Pull request: no
Labels: name: bug

### System Info

![Screenshot 2024-08-29 094331](https://github.com/user-attachments/assets/f14d605a-98fa-4b5d-bd7a-da704bdb8ab3)


### Who can help?

@ArthurZucker, @SunMarc

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```py
!pip install xformers==0.0.27.post2 diffusers==0.30.1 transformers==4.42.4 accelerate==0.33.0 sentencepiece==0.2.0 moviepy==1.0.3
!apt -y install -qq aria2

!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/vdo/CogVideoX-5b/raw/main/scheduler/scheduler_config.json -d /content/CogVideoX-5b/scheduler -o scheduler_config.json
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/vdo/CogVideoX-5b/raw/main/text_encoder/config.json -d /content/CogVideoX-5b/text_encoder -o config.json
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/vdo/CogVideoX-5b/resolve/main/text_encoder/model-00001-of-00002.safetensors -d /content/CogVideoX-5b/text_encoder -o model-00001-of-00002.safetensors
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/vdo/CogVideoX-5b/resolve/main/text_encoder/model-00002-of-00002.safetensors -d /content/CogVideoX-5b/text_encoder -o model-00002-of-00002.safetensors
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/vdo/CogVideoX-5b/raw/main/text_encoder/model.safetensors.index.json -d /content/CogVideoX-5b/text_encoder -o model.safetensors.index.json
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/vdo/CogVideoX-5b/raw/main/tokenizer/added_tokens.json -d /content/CogVideoX-5b/tokenizer -o added_tokens.json
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/vdo/CogVideoX-5b/raw/main/tokenizer/special_tokens_map.json -d /content/CogVideoX-5b/tokenizer -o special_tokens_map.json
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/vdo/CogVideoX-5b/resolve/main/tokenizer/spiece.model -d /content/CogVideoX-5b/tokenizer -o spiece.model
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/vdo/CogVideoX-5b/raw/main/tokenizer/tokenizer_config.json -d /content/CogVideoX-5b/tokenizer -o tokenizer_config.json
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/vdo/CogVideoX-5b/raw/main/transformer/config.json -d /content/CogVideoX-5b/transformer -o config.json
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/vdo/CogVideoX-5b/resolve/main/transformer/diffusion_pytorch_model-00001-of-00002.safetensors -d /content/CogVideoX-5b/transformer -o diffusion_pytorch_model-00001-of-00002.safetensors
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/vdo/CogVideoX-5b/resolve/main/transformer/diffusion_pytorch_model-00002-of-00002.safetensors -d /content/CogVideoX-5b/transformer -o diffusion_pytorch_model-00002-of-00002.safetensors
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/vdo/CogVideoX-5b/raw/main/transformer/diffusion_pytorch_model.safetensors.index.json -d /content/CogVideoX-5b/transformer -o diffusion_pytorch_model.safetensors.index.json
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/vdo/CogVideoX-5b/raw/main/vae/config.json -d /content/CogVideoX-5b/vae/ -o config.json
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/vdo/CogVideoX-5b/resolve/main/vae/diffusion_pytorch_model.safetensors -d /content/CogVideoX-5b/vae -o diffusion_pytorch_model.safetensors
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/vdo/CogVideoX-5b/raw/main/configuration.json -d /content/CogVideoX-5b -o configuration.json
!aria2c --console-log-level=error -c -x 16 -s 16 -k 1M https://huggingface.co/vdo/CogVideoX-5b/raw/main/model_index.json -d /content/CogVideoX-5b -o model_index.json

import torch
from diffusers import AutoencoderKLCogVideoX, CogVideoXTransformer3DModel, CogVideoXDDIMScheduler
from transformers import T5EncoderModel, T5Tokenizer

transformer = CogVideoXTransformer3DModel.from_pretrained("/content/CogVideoX-5b", subfolder="transformer", torch_dtype=torch.float8_e4m3fn)
text_encoder = T5EncoderModel.from_pretrained("/content/CogVideoX-5b", subfolder="text_encoder", torch_dtype=torch.float8_e4m3fn)
vae = AutoencoderKLCogVideoX.from_pretrained("/content/CogVideoX-5b", subfolder="vae", torch_dtype=torch.float8_e4m3fn)
scheduler = CogVideoXDDIMScheduler.from_pretrained("/content/CogVideoX-5b", subfolder="scheduler")
tokenizer = T5Tokenizer.from_pretrained("/content/CogVideoX-5b", subfolder="tokenizer")

pipe = CogVideoXPipeline(transformer=transformer, text_encoder=text_encoder, vae=vae, tokenizer=tokenizer, scheduler=scheduler)
```

### Expected behavior

why does `torch_dtype=torch.float8_e4m3fn` work with transformer and vae but not with text_encoder?

Comments:
Thanks for your issue @camenduru! cc @SunMarc is the right person to help you out.

Hey @camenduru, it is not possible load a model with a torch_dtype = torch.float8_e4m3fn. The accepted values are torch.float32, torch.bfloat16 and torch.float16. Did you find a tutorial that does this ? 

Hey @SunMarc, thanks for the answer. Why is it not possible?

Models are not trained to be used in that dtype. However, if you want to use 8bit model, I advise you to load in fp16 then apply a quantization method like bnb to get the 8-bit or 4-bit model