noamaanMulla-03/transformers-issues huggingface dataset:
`load_in_4bit=True` works only with models in `safetensors` format
Author: danielkorat
Pull request: no

### System Info

Hi,

When testing on `Google Colab (Free Tier T4 GPU)`, this code crashes with RAM OOM [(Notebook)](https://colab.research.google.com/drive/1zAzdcH_KRQuc_0zWBEzYuaV1h4ERgzPy?usp=sharing):

```python
AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1", load_in_4bit=True, device_map="auto")
```

![image](https://github.com/huggingface/transformers/assets/32893314/11f62cdc-b94c-4e68-9783-c4d25904f194)

However, when loading the safetensors revision, it works:

```python
AutoModelForCausalLM.from_pretrained("mistralai/Mistral-7B-v0.1", load_in_4bit=True, device_map="auto", revision="/refs/pr/91")
```

```bash
transformers==4.35.2
accelerate==0.24.1
bitsandbytes==0.41.2.post2
```

cc @ArthurZucker @younesbelkada 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Colab link that reproduces the problem: [Notebook](https://colab.research.google.com/drive/1zAzdcH_KRQuc_0zWBEzYuaV1h4ERgzPy?usp=sharing)



### Expected behavior

Successful loading of the model onto GPU.

Comments:
Hey! Thanks for reporting. I would not be surprised that `safetensors` supports loading each parameters indivudally, casting them which is not supported by torch.  I think that using smaller shards is usually what we do in this case, feel free to use: https://huggingface.co/ybelkada/Mistral-7B-v0.1-bf16-sharded

Thanks @ArthurZucker 
Since this is a very general use case, and also happens out of the blue for many existing models like `Intel/neural-chat-7b-v3-1` as well, I think there should be some fix or at least a warning. Thanks

We changed the default sharding size to make sure colab supports this. 

Hi @danielkorat 
As @ArthurZucker stated we changed the default max shard size to support google colab inference easily, for previous models, unfortunately one needs to manually shard the model and push it on the hub under a new model id. 

I see.
I solved it in my case by merging the safetensors PR, under the same model_id (`Intel/neural-chat-7b-v3-1`). 
I think users like me and others I've seen might be running into this issue right now in Colab, without any explanation of the cause. Would be nice if there was at least a warning when loading very large shards.
@younesbelkada 

Hi @danielkorat 
Thanks! 
Hmmm I think this might introduce a lot of verbosity in transformers as this would only be relevant in the case people use google colab

Then you can probably close this, thanks.

Thank you @danielkorat !
Whisper: Language detection not working after setting a language once
Author: brunjo
Pull request: no

### System Info

- `transformers` version: 4.36.0.dev0
- Platform: Linux-6.2.0-1018-gcp-x86_64-with-glibc2.31
- Python version: 3.11.6
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.0
- Accelerate version: 0.24.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.1+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes, T4 GPU
- Using distributed or parallel set-up in script?: No

### Who can help?

@sanchit-gandhi 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When calling the Whisper pipeline with a language (eg `generate_kwargs={"language": "french"}`) the language is set for all future pipeline calls even when setting it explicitly to `None` (ie `generate_kwargs={"language": None}`).

```python
# Set up `pipe` as described at https://huggingface.co/openai/whisper-large-v3#usage

# This works as expected:
pipe("brownfox.mp3", generate_kwargs={"language": None})
# > {'text': ' The quick brown fox jumps over the lazy dog.'}

pipe("brownfox.mp3", generate_kwargs={"language": "french"})
# > {'text': ' Le foie vert rapide tombe sur le chien loup.'}

# Here the language should be auto-detected as English but I'm getting a response in French
pipe("brownfox.mp3", generate_kwargs={"language": None})
# > {'text': ' Le foie vert rapide tombe sur le chien loup.'}
```

Audio used in this example: [brownfox.mp3](https://output.lemonfox.ai/brownfox.mp3)

### Expected behavior

I would expect the language to be auto-detected when `language` is not set even when `language` was set in a previous call.

Comments:
cc @sanchit-gandhi and @ylacombe 

I've found a workaround: Running `del pipe.model.generation_config.language` before calling the `pipe` method resets the language parameter and fixes the issue.
Documentation: Spanish translation of perplexity.mdx
Author: aaronjimv
Pull request: yes

# What does this PR do?

Add the Spanish version of perplexity.mdx to transformers/docs/source/es

Fixes #15947 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@omarespejel @sgugger @osanseviero 

Comments:
Hello. I am open to any feedback. Thanks.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27807). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[`Docs`] Update broken image on fused modules
Author: younesbelkada
Pull request: yes

# What does this PR do?

As per title, as pointed out by @SunMarc offline



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27856). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update CUDA versions for DeepSpeed
Author: muellerzr
Pull request: yes

# What does this PR do?

DeepSpeed tests are failing because they require higher CUDA versions


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ydshieh 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27853). All of your documentation changes will be reflected on that endpoint.
[docstring] Fix docstring for AltCLIPVisionConfig, AltCLIPTextConfig and AltCLIPConfig
Author: AksharGoyal
Pull request: no

              I will take AltCLIPVisionConfig, AltCLIPTextConfig

_Originally posted by @AksharGoyal in https://github.com/huggingface/transformers/issues/26638#issuecomment-1769186608_
            

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Cache round 2
Author: gante
Pull request: yes

# What does this PR do?

Adds indexing to the cache, as `generate` relies on things like `past_key_values[0][0].shape` (the first set of indexes being 0 for the attention `key` and 1 for the attention `value`).

After this change, the following snippet produces the same result as before:
```py
from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed
import torch

set_seed(0)

tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf", torch_dtype=torch.float16, device_map="auto")

inputs = tokenizer(["The best color is"], return_tensors="pt").to(model.device)
gen_out = model.generate(
    **inputs,
    do_sample=False,
    max_new_tokens=20,
    use_legacy_cache=False,
    num_beams=2,
    num_return_sequences=2,
)
print(tokenizer.batch_decode(gen_out, skip_special_tokens=True))
```

Comments:
Avoid class attribute `_keep_in_fp32_modules` being modified
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix #25910

Currently, for `InstructBlip` which can combine/use different architecture as component, we have the following block

https://github.com/huggingface/transformers/blob/abd253103478b21faafb7c9a6e7a1a7d1effe757/src/transformers/models/instructblip/modeling_instructblip.py#L1280-L1281

in order to extent the class attribute `_keep_in_fp32_modules` of `InstructBlipPreTrainedModel`. This modification of class attribute causes some issue:
- loading `Salesforce/instructblip-flan-t5-xl` changes `_keep_in_fp32_modules` from `[]` to `["wo"]`, because the language model is a `T5PreTrainedModel` which has `_keep_in_fp32_modules = ["wo"]`.
  - Then loading `Salesforce/instructblip-vicuna-7b` will keep `qformer.embeddings.word_embeddings.weight` in `fp32`, due to a bug before the fix #26589.
- If we don't load `flan-t5` but just `vicuna`, `qformer.embeddings.word_embeddings.weight` will be in `fp16`.
and the weight values are slightly different -> outputs are different -> CI failure.

**Modifying a class attribute is bad** in general, and we have failing test. This CI tries to incorporate components' in `_keep_in_fp32_modules` without modifying the `_keep_in_fp32_modules` in the parent model class.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26433). All of your documentation changes will be reflected on that endpoint.

This still looks great, what is missing to merge it @ydshieh ?

nothing, just didn't get an approval from core maintainers. Will ping one.

Before merge this PR (if being approved), I have to apply the change to some recently modified files.

> If _keep_in_fp32_modules is just an instance attribute instead of a class attribute

I agree, but this implies that we have to put this attribute in each concrete class (e.g. BertForXXX), rather than just, say, `BertPretrainedModel`. And in most case, the value is the same.


There is also an usage of `cls._keep_in_fp32_modules` so far

```python
            # Check if `_keep_in_fp32_modules` is not None
            use_keep_in_fp32_modules = (cls._keep_in_fp32_modules is not None) and (
                torch_dtype == torch.float16 or load_in_4bit or load_in_8bit
            )
```
where an instance is not created yet.

No you can put it in BerPretrainedModel's `__init__`. And if the check relies on the class, the class can still have one I think the init just overwrites it

Yes, that works well, thanks.
[`Flash Attention 2`] Add flash attention 2 for GPT-Neo-X
Author: younesbelkada
Pull request: yes

# What does this PR do?

Adds flash attention support for GPT-Neo-X

Fixes: https://github.com/huggingface/transformers/issues/26444



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26463). All of your documentation changes will be reflected on that endpoint.

Any plans on completing this or should someone else pick it up? For what it's worth, this implementation is working very well for me 👍 

cc @amyeroberts let me know if I need to address anything else in this PR! 

Checking on the progress here. What's the ETA on merging this with the main branch? Thanks!
Translating en/model_doc folder docs to Japanese(from `blip` to `clap`) 🇯🇵
Author: rajveer43
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #27669


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?


Documentation: @stevhliu and @MKhalusova



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27673). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> LGTM! Can you try running `make fixup` and `pip uninstall black && pip install -U ruff==0.1.5` to fix the failing CI test?

sure.
[i18n-jp] Translating `en/model_doc` folder docs to Japanese 🇯🇵
Author: rajveer43
Pull request: no
Labels: name: WIP

<!--
Note: Please search to see if an issue already exists for the language you are trying to translate.
-->

Hi!

Let's bring the documentation to all the japanese-speaking community 🌐 (currently 0 out of 267 complete)

Who would want to translate? Please follow the 🤗 [TRANSLATING guide](https://github.com/huggingface/transformers/blob/main/docs/TRANSLATING.md). Here is a list of the files ready for translation. Let us know in this issue if you'd like to translate any, and we'll add your name to the list.

Some notes:

* Please translate using an informal tone (imagine you are talking with a friend about transformers 🤗).
* Please translate in a gender-neutral way.
* Add your translations to the folder called `ja` inside the [source folder](https://github.com/huggingface/transformers/tree/main/docs/source).
* Register your translation in `<ja/_toctree.yml`; please follow the order of the [English version](https://github.com/huggingface/transformers/blob/main/docs/source/en/_toctree.yml).
* Once you're finished, open a pull request and tag this issue by including #issue-number in the description, where issue-number is the number of this issue. Please ping @stevhliu and @MKhalusova for review.
* 🙋 If you'd like others to help you with the translation, you can also post in the 🤗 [forums](https://discuss.huggingface.co/).

## Model_doc section

- [ ] [blip-2.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_doc/blip-2.md) #27673 
- [ ] [bloom.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_doc/bloom.md) #27673 
- [ ] [bort.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_doc/bort.md). #27673 
- [ ] [bridgetower.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_doc/bridgetower.md). #27673 
- [ ] [bros.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_doc/bros.md). #27673 
- [ ] [byt5.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_doc/byt5.md). #27673 
- [ ] [camembert.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_doc/camembert.md). #27673 
- [ ] [canine.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_doc/canine.md). #27673 
- [ ] [chinese_clip.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_doc/chinese_clip.md). #27673 
- [ ] [clap.md](https://github.com/huggingface/transformers/blob/main/docs/source/en/model_doc/clap.md). #27673 




Comments:
Update `VitDetModelTester.get_config` to use `pretrain_image_size`
Author: ydshieh
Pull request: yes

# What does this PR do?

`torch.nn.init.trunc_normal_` has strange behavior (at least, on CPU) if the input tensor is large, say `size=(4096, 32)`, and sometimes produces very large item in a tensor, even if `mean=0 and std=1e10`.

This PR updates `VitDetModelTester.get_config` to use `pretrain_image_size=self.image` (224 before --> 30 now), so the `position_embeddings` is of much smaller size. This is always welcome no matter if we have the above issue or not.

I will open an issue in pytorch repository, but I think the issue is well known ...

Comments:
The failing test is from #27729. I already ping NielsRogge offline.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27831). All of your documentation changes will be reflected on that endpoint.

I don't even know what 🧼  means, but I will merge.
Add Mistral Models to Flax
Author: kiansierra
Pull request: no
Labels: name: Feature request

### Feature request

I would like to implement the ~~Llama~~ Mistral model in flax

### Motivation

I've been trying to get familiar with jax and as such I started migrating the llama model, and I think I am at a point where both models are quite comparable in outcome

### Your contribution

Yes I could submit a PR with the model implementation

Comments:
I think this could be interesting! Feel free to open a PR and ping @sanchit-gandhi 😉 

Hey @kiansierra - there's already a PR for Flax LLaMA that is pretty much ready to be merged: https://github.com/huggingface/transformers/pull/24587 Feel free to check it out!

But we'd love contributions for other LLM's in the library where there's only PyTorch support and not Flax 🤗 If there are particular checkpoints on the HF Hub that you see getting a lot of usage (downloads) where there's only PyTorch support but not Flax, definitely let us know here and we can get going with a PR! 🚀

Thansk for the Heads up @sanchit-gandhi, I'll see if there is any other model I think I can add to Flax and tag you on the next issue

Oups I even reviewed the PR 😅 sorry @kiansierra 🤗 

@kiansierra sorry to scoop Flax Llama from you! If you want any suggestions, I think [Mistral](https://github.com/huggingface/transformers/blob/main/src/transformers/models/mistral/modeling_mistral.py) is a pretty popular model right now without a Flax port.

Hey no worries, I think I will give Mistral a go, it seems some of the work can be ported

Happy to see that a couple of people are interested in porting these models to flax! I was also interested in contributing! Is there any other model that would be interesting? On a side note: I guess flash-attention only works for the pytorch models atm (?) Is there any fundamental reason why porting the flash-attention implementation to jax would be difficult?

hello, guys I have created both llama and mistral models in flax if you want you can use them [modelling_mistral_flax.py](https://github.com/erfanzar/EasyDeL/blob/main/lib/python/EasyDel/modules/mistral/modelling_mistral_flax.py)

Yes Flash Attention relies on dispatching optimised CUDA kernels, which as far as I'm aware haven't been implemented in JAX. You could look into Pallas and see if someone's written Flash Attention kernels for JAX using this library? https://jax.readthedocs.io/en/latest/pallas/design.html

Indeed there's an effort to write FlashAttention in Pallas, https://github.com/google/jax/blob/main/jax/experimental/pallas/ops/attention.py although it's still a work in progress https://github.com/google/jax/pull/17328 . @sanchit-gandhi I'd be happy to try to port another model. For example, Yarn-Mistral seems to have some traction, though it's not part of the transformers library atm. Any other suggestions are welcome!
Converting Megatron_T5 to HF_T5
Author: WenzhengZhang
Pull request: no
Labels: name: Good Second Issue

Currently both [Megatron_BERT conversion](https://github.com/huggingface/transformers/blob/main/src/transformers/models/megatron_bert/convert_megatron_bert_checkpoint.py) and [Megatron_GPT conversion](https://github.com/huggingface/transformers/blob/main/src/transformers/models/megatron_gpt2/convert_megatron_gpt2_checkpoint.py) are supported. Do you have any plan to support T5 models conversion?

Comments:
Hello guys,
I saw you closed this issue.
Did you find any way to convert megatron t5 <-> hf T5?

I would be interested too.
Did you find any solution?

any potential solutions?

Don't think this is supported yet, but feel free to open a pr if you want! 🤗
`ACT2FN` usage
Author: ariG23498
Pull request: no

In the code below we have a function to retrieve the correct activation function from the string.
https://github.com/huggingface/transformers/blob/235e5d4991e8a0984aa78db91087b49622c7740e/src/transformers/activations.py#L224-L228

I have noticed some models to use this function and others to directly import `ACT2FN` and use it as a dictionary.

IMO the function is redundant and importing the dictionary directly to scripts is enough. Curious to hear other opinions.

Comments:
Hey! This seems to be related to backwards compatibility but we favor using the ACT2FN directly 

Thanks for the prompt response. You can close this issue if you would like.