expected scalar type Float but found Half
Author: Aniruddha-JU
Pull request: no

### System Info

Install latest trsansformer, accelarate, and butsandbytes.

### Who can help?

from transformers import T5Tokenizer, T5ForConditionalGeneration
@sgugger 


### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

tokenizer = T5Tokenizer.from_pretrained("google/flan-t5-xxl")
model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-xxl", device_map="auto", load_in_8bit=True)

input_text = "translate English to German: How old are you?"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")

outputs = model.generate(input_ids)
print(tokenizer.decode(outputs[0]))

### Expected behavior

error solve

Comments:
Model successfully loaded, but during model.generate line throwing the error.


Please copy-paste the whole traceback, as is asked in the issue templare.
cc @ArthurZucker and @younesbelkada 

@ArthurZucker @younesbelkada @sgugger 
```python
RuntimeError                              Traceback (most recent call last)
Cell In[35], line 1
----> 1 outputs = model.generate(input_ids)

File /usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py:115, in context_decorator.<locals>.decorate_context(*args, **kwargs)
    112 @functools.wraps(func)
    113 def decorate_context(*args, **kwargs):
    114     with ctx_factory():
--> 115         return func(*args, **kwargs)

File /usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:1486, in GenerationMixin.generate(self, inputs, generation_config, logits_processor, stopping_criteria, prefix_allowed_tokens_fn, synced_gpus, assistant_model, streamer, negative_prompt_ids, negative_prompt_attention_mask, **kwargs)
   1478         logger.warning(
   1479             "A decoder-only architecture is being used, but right-padding was detected! For correct "
   1480             "generation results, please set `padding_side='left'` when initializing the tokenizer."
   1481         )
   1483 if self.config.is_encoder_decoder and "encoder_outputs" not in model_kwargs:
   1484     # if model is encoder decoder encoder_outputs are created
   1485     # and added to `model_kwargs`
-> 1486     model_kwargs = self._prepare_encoder_decoder_kwargs_for_generation(
   1487         inputs_tensor, model_kwargs, model_input_name
   1488     )
   1490 # 5. Prepare `input_ids` which will be used for auto-regressive generation
   1491 if self.config.is_encoder_decoder:

File /usr/local/lib/python3.10/dist-packages/transformers/generation/utils.py:655, in GenerationMixin._prepare_encoder_decoder_kwargs_for_generation(self, inputs_tensor, model_kwargs, model_input_name)
    653 encoder_kwargs["return_dict"] = True
    654 encoder_kwargs[model_input_name] = inputs_tensor
--> 655 model_kwargs["encoder_outputs"]: ModelOutput = encoder(**encoder_kwargs)
    657 return model_kwargs

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1502, in Module._wrapped_call_impl(self, *args, **kwargs)
   1500     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1501 else:
-> 1502     return self._call_impl(*args, **kwargs)

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1511, in Module._call_impl(self, *args, **kwargs)
   1506 # If we don't have any hooks, we want to skip the rest of the logic in
   1507 # this function, and just call forward.
   1508 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1509         or _global_backward_pre_hooks or _global_backward_hooks
   1510         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1511     return forward_call(*args, **kwargs)
   1512 # Do not call functions when jit is used
   1513 full_backward_hooks, non_full_backward_hooks = [], []

File /usr/local/lib/python3.10/dist-packages/transformers/models/t5/modeling_t5.py:1123, in T5Stack.forward(self, input_ids, attention_mask, encoder_hidden_states, encoder_attention_mask, inputs_embeds, head_mask, cross_attn_head_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)
   1110     layer_outputs = checkpoint(
   1111         create_custom_forward(layer_module),
   1112         hidden_states,
   (...)
   1120         None,  # past_key_value is always None with gradient checkpointing
   1121     )
   1122 else:
-> 1123     layer_outputs = layer_module(
   1124         hidden_states,
   1125         attention_mask=extended_attention_mask,
   1126         position_bias=position_bias,
   1127         encoder_hidden_states=encoder_hidden_states,
   1128         encoder_attention_mask=encoder_extended_attention_mask,
   1129         encoder_decoder_position_bias=encoder_decoder_position_bias,
   1130         layer_head_mask=layer_head_mask,
   1131         cross_attn_layer_head_mask=cross_attn_layer_head_mask,
   1132         past_key_value=past_key_value,
   1133         use_cache=use_cache,
   1134         output_attentions=output_attentions,
   1135     )
   1137 # layer_outputs is a tuple with:
   1138 # hidden-states, key-value-states, (self-attention position bias), (self-attention weights), (cross-attention position bias), (cross-attention weights)
   1139 if use_cache is False:

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1502, in Module._wrapped_call_impl(self, *args, **kwargs)
   1500     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1501 else:
-> 1502     return self._call_impl(*args, **kwargs)

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1511, in Module._call_impl(self, *args, **kwargs)
   1506 # If we don't have any hooks, we want to skip the rest of the logic in
   1507 # this function, and just call forward.
   1508 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1509         or _global_backward_pre_hooks or _global_backward_hooks
   1510         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1511     return forward_call(*args, **kwargs)
   1512 # Do not call functions when jit is used
   1513 full_backward_hooks, non_full_backward_hooks = [], []

File /usr/local/lib/python3.10/dist-packages/accelerate/hooks.py:165, in add_hook_to_module.<locals>.new_forward(*args, **kwargs)
    163         output = old_forward(*args, **kwargs)
    164 else:
--> 165     output = old_forward(*args, **kwargs)
    166 return module._hf_hook.post_forward(module, output)

File /usr/local/lib/python3.10/dist-packages/transformers/models/t5/modeling_t5.py:695, in T5Block.forward(self, hidden_states, attention_mask, position_bias, encoder_hidden_states, encoder_attention_mask, encoder_decoder_position_bias, layer_head_mask, cross_attn_layer_head_mask, past_key_value, use_cache, output_attentions, return_dict)
    692 else:
    693     self_attn_past_key_value, cross_attn_past_key_value = None, None
--> 695 self_attention_outputs = self.layer[0](
    696     hidden_states,
    697     attention_mask=attention_mask,
    698     position_bias=position_bias,
    699     layer_head_mask=layer_head_mask,
    700     past_key_value=self_attn_past_key_value,
    701     use_cache=use_cache,
    702     output_attentions=output_attentions,
    703 )
    704 hidden_states, present_key_value_state = self_attention_outputs[:2]
    705 attention_outputs = self_attention_outputs[2:]  # Keep self-attention outputs and relative position weights

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1502, in Module._wrapped_call_impl(self, *args, **kwargs)
   1500     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1501 else:
-> 1502     return self._call_impl(*args, **kwargs)

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1511, in Module._call_impl(self, *args, **kwargs)
   1506 # If we don't have any hooks, we want to skip the rest of the logic in
   1507 # this function, and just call forward.
   1508 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1509         or _global_backward_pre_hooks or _global_backward_hooks
   1510         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1511     return forward_call(*args, **kwargs)
   1512 # Do not call functions when jit is used
   1513 full_backward_hooks, non_full_backward_hooks = [], []

File /usr/local/lib/python3.10/dist-packages/accelerate/hooks.py:165, in add_hook_to_module.<locals>.new_forward(*args, **kwargs)
    163         output = old_forward(*args, **kwargs)
    164 else:
--> 165     output = old_forward(*args, **kwargs)
    166 return module._hf_hook.post_forward(module, output)

File /usr/local/lib/python3.10/dist-packages/transformers/models/t5/modeling_t5.py:601, in T5LayerSelfAttention.forward(self, hidden_states, attention_mask, position_bias, layer_head_mask, past_key_value, use_cache, output_attentions)
    591 def forward(
    592     self,
    593     hidden_states,
   (...)
    599     output_attentions=False,
    600 ):
--> 601     normed_hidden_states = self.layer_norm(hidden_states)
    602     attention_output = self.SelfAttention(
    603         normed_hidden_states,
    604         mask=attention_mask,
   (...)
    609         output_attentions=output_attentions,
    610     )
    611     hidden_states = hidden_states + self.dropout(attention_output[0])

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1502, in Module._wrapped_call_impl(self, *args, **kwargs)
   1500     return self._compiled_call_impl(*args, **kwargs)  # type: ignore[misc]
   1501 else:
-> 1502     return self._call_impl(*args, **kwargs)

File /usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py:1511, in Module._call_impl(self, *args, **kwargs)
   1506 # If we don't have any hooks, we want to skip the rest of the logic in
   1507 # this function, and just call forward.
   1508 if not (self._backward_hooks or self._backward_pre_hooks or self._forward_hooks or self._forward_pre_hooks
   1509         or _global_backward_pre_hooks or _global_backward_hooks
   1510         or _global_forward_hooks or _global_forward_pre_hooks):
-> 1511     return forward_call(*args, **kwargs)
   1512 # Do not call functions when jit is used
   1513 full_backward_hooks, non_full_backward_hooks = [], []

File /usr/local/lib/python3.10/dist-packages/accelerate/hooks.py:165, in add_hook_to_module.<locals>.new_forward(*args, **kwargs)
    163         output = old_forward(*args, **kwargs)
    164 else:
--> 165     output = old_forward(*args, **kwargs)
    166 return module._hf_hook.post_forward(module, output)

File /usr/local/lib/python3.10/dist-packages/apex/normalization/fused_layer_norm.py:386, in FusedRMSNorm.forward(self, input)
    383     return manual_rms_norm(input, self.normalized_shape, self.weight, self.eps)
    385 if self.elementwise_affine:
--> 386     return fused_rms_norm_affine(input, self.weight, self.normalized_shape, self.eps)
    387 else:
    388     return fused_rms_norm(input, self.normalized_shape, self.eps)

File /usr/local/lib/python3.10/dist-packages/apex/normalization/fused_layer_norm.py:189, in fused_rms_norm_affine(input, weight, normalized_shape, eps)
    187 args = _cast_if_autocast_enabled(input, weight, normalized_shape, eps)
    188 with torch.cuda.amp.autocast(enabled=False):
--> 189     return FusedRMSNormAffineFunction.apply(*args)

File /usr/local/lib/python3.10/dist-packages/torch/autograd/function.py:506, in Function.apply(cls, *args, **kwargs)
    503 if not torch._C._are_functorch_transforms_active():
    504     # See NOTE: [functorch vjp and autograd interaction]
    505     args = _functorch.utils.unwrap_dead_wrappers(args)
--> 506     return super().apply(*args, **kwargs)  # type: ignore[misc]
    508 if cls.setup_context == _SingleLevelFunction.setup_context:
    509     raise RuntimeError(
    510         'In order to use an autograd.Function with functorch transforms '
    511         '(vmap, grad, jvp, jacrev, ...), it must override the setup_context '
    512         'staticmethod. For more details, please see '
    513         '[https://pytorch.org/docs/master/notes/extending.func.html](https://pytorch.org/docs/master/notes/extending.func.html%3C/span%3E%3Cspan) style="color:rgb(175,0,0)">')

File /usr/local/lib/python3.10/dist-packages/apex/normalization/fused_layer_norm.py:69, in FusedRMSNormAffineFunction.forward(ctx, input, weight, normalized_shape, eps)
     67 input_ = input.contiguous()
     68 weight_ = weight.contiguous()
---> 69 output, invvar = fused_layer_norm_cuda.rms_forward_affine(
     70     input_, ctx.normalized_shape, weight_, ctx.eps)
     71 ctx.save_for_backward(input_, weight_, invvar)
     72 return output

RuntimeError: expected scalar type Float but found Half
```

Hi @Aniruddha-JU 
Thanks for the issue, this is know that T5 + apex + int8 does currently not work 
Check out the related issues https://github.com/huggingface/transformers/issues/21656 / https://github.com/huggingface/transformers/issues/21391
The fix now is to uninstall apex from your env

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.