noamaanMulla-03/transformers-issues huggingface dataset:
Add sliding window attention to sdpa in mistral
Author: ehuaa
Pull request: no
Labels: name: Good Second Issue

### Feature request

https://github.com/huggingface/transformers/blob/main/src/transformers/models/mistral/modeling_mistral.py#L1006-L1023
![image](https://github.com/huggingface/transformers/assets/5137359/9601a5d2-cf9f-4ef6-a0ab-047a8cd7f1cd)

In the code listed above, the latest version of transformers cannot use sliding window feature in mistral model.
I doubt that the reason is you mentioned above,
https://github.com/huggingface/transformers/blob/main/src/transformers/models/mistral/modeling_mistral.py#L687-L688
![image](https://github.com/huggingface/transformers/assets/5137359/997cd770-e17f-4eb4-997b-fe65f30ddc85)
And this issue in PyTorch makes you bugged with custom attn_mask like sliding window attention mask.
https://github.com/pytorch/pytorch/issues/112577

While this issue has been fixed since torch 2.2.0, and it has been released two weeks ago, can you add this feature back to sdpa kernel in mistral?

### Motivation

I cannot use sliding window with sdpa right now, cause my gpu card is V100, i cannot work with flashattention2.

### Your contribution

I think we can pass sliding_window param to _prepare_4d_causal_attention_mask_for_sdpa function.

Comments:
cc @fxmarty 

Hi, thank you for the suggestion, SDPA support for mistral was added by @ArthurZucker in https://github.com/huggingface/transformers/pull/28133, maybe he has more insight.

I think it comes down to just adding `sliding_window` to the call for `_prepare_4d_causal_attention_mask_for_sdpa` yes. Would you like to open a PR?

> I think it comes down to just adding `sliding_window` to the call for `_prepare_4d_causal_attention_mask_for_sdpa` yes. Would you like to open a PR?

Sure，and i'll open a PR later in this week

any plan for pr?

#29407 should fix this issue

@ArthurZucker Oh you are right. Thanks.

Fixed in https://github.com/huggingface/transformers/pull/30127
Fix SpeechT5 forward docstrings
Author: ylacombe
Pull request: yes

# What does this PR do?

SpeechT5ForTextToSpeech.forward code snippet [was failing](https://github.com/huggingface/transformers/actions/runs/8659540805/job/23745789410), this fixes it!

cc @ydshieh !

Comments:
Looks reasonable to me! Thanks!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30287). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
FIX / AWQ: Fix failing exllama test
Author: younesbelkada
Pull request: yes

Fixes the flaky slow test with AWQ + exllama-v2 - from an offline discussion with @SunMarc 

> This test is flaky. The output changes depending on the run ( works on single, fail on multi and sometimes it is the opposite) . The fix would be to accept both outputs.

cc @ydshieh @SunMarc 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30288). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix test `ExamplesTests::test_run_translation`
Author: ydshieh
Pull request: yes

# What does this PR do?

`ExamplesTests::test_run_translation`.

@gante  broke this in #29600 by adding a new check
```python
        raise ValueError(
            f"`--max_source_length` is set to {data_args.max_source_length}, but the model only has"
            f" {model.config.max_position_embeddings} position encodings. Consider either reducing"
            f" `--max_source_length` to {model.config.max_position_embeddings} or using a model with larger position "
            "embeddings"
        )
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30281). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix `Fatal Python error: Bus error` in `ZeroShotAudioClassificationPipelineTests`
Author: ydshieh
Pull request: yes

# What does this PR do?

Same fix as in [here](https://github.com/huggingface/transformers/commit/cbc2cc187aeefd85aac8bb1660d5874ddc1164d2#diff-185d4b35e509f8dc00eb38307d0afc56ea5d1ae526c499dd63b157179cce2e14)

Currently, torch pipeline tests timeout on GitHub Actions (after `6 hours`), see [here](https://github.com/huggingface/transformers/actions/runs/8682815917/job/23807891346). This starts on March 21 daily CI run - not clear what changes caused it, but I see `Fatal Python error: Bus error` is in March 21's log but not in March 20.

And with this PR, the run seems to be quite fast, [40 mins in this run](https://github.com/huggingface/transformers/actions/runs/8716531699/job/23910217304)

(this PR only fixes the timeout issue, not all the failing tests in torch pipeline job)


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30283). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
FIX: Fix corner-case issue with the important models workflow
Author: younesbelkada
Pull request: yes

As per title

Before this PR, there was a bug when `models/xxx files where modified because a check for empty arrays on the GitHub action was being bypassed leading to the action being run unexpectedly such as: https://github.com/huggingface/transformers/actions/runs/8688185234 - this PR fixes it

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30212). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Tested it and it worked on [1a26ff3](https://github.com/huggingface/transformers/pull/30212/commits/1a26ff36e663be0eca774d4affad59fb95a98f46) and I made sure we still have the previous behaviour [6a527d3](https://github.com/huggingface/transformers/pull/30212/commits/6a527d3d07653bdfe905d132a3b3ff137c608d40) this is ready for review !

@younesbelkada Could you include in the PR description what the corner case is? I can see this is a fix but it's not clear what 's being addressed 

Thanks @amyeroberts ! Sorry, yes - I just added some description, let me know what do you think - here is also an example of that corner case: https://huggingface.slack.com/archives/C06L2SGMEEA/p1713186513863159 

Thanks for the review ! 
let me know if https://github.com/huggingface/transformers/pull/30212#discussion_r1566008487 makes sense ! I will let you merge if you agree with the explanation ! 🙏 Thanks again !
Fix slow tests for important models to be compatible with A10 runners
Author: ydshieh
Pull request: yes

# What does this PR do?

This PR is the same as #29805 with some final changes. #29805 is closed, and we will merge this one without waiting @younesbelkada being back to the office.

I rebased the PR on a more recent `main`.

 No need to go over again I think. Just check [this comment](https://github.com/huggingface/transformers/pull/29905#issuecomment-2022936196) (if you think necessary)

Comments:
Although it is newly opened (despite with tiny changes), formally let me get a ✅ approval before merge.

here are the changes: mostly, they are just some name changes

[1](https://github.com/huggingface/transformers/pull/29905/commits/ad041bb32870c7ca3a104e6d47b6dec3dbc6cd1a)
[2](https://github.com/huggingface/transformers/pull/29905/commits/b40d111c6a20bb4ebe147241fcf5e5f82b21a7a5)
[3](https://github.com/huggingface/transformers/pull/29905/commits/f67d34fff7865926fb6033a83862389c5017c1c2)


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29905). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

The only failing test is 

> FAILED tests/models/whisper/test_modeling_whisper.py::WhisperModelTest::test_generate_with_prompt_ids_max_length - IndexError: index -1 is out of bounds for dimension 1 with size 0

which is irrelevant to this PR, and #30018 is on it.

@younesbelkada There are still some test failures

https://github.com/huggingface/transformers/actions/runs/8542383405/job/23404051565

tests are all green: https://github.com/huggingface/transformers/actions/runs/8613471616/job/23604832895

Thank you @younesbelkada again!
FIX: Fix push important models CI
Author: younesbelkada
Pull request: yes

# What does this PR do?

Remove duplicated `runs-on`

cc @amyeroberts @ydshieh 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30291). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
'CLIPEncoder' object has no attribute '_gradient_checkpointing_func'
Author: TideDra
Pull request: no
Labels: name: Multimodal

### System Info

- `transformers` version: 4.39.1
- Platform: Linux-5.15.0-1041-azure-x86_64-with-glibc2.35
- Python version: 3.10.14
- Huggingface_hub version: 0.22.2
- Safetensors version: 0.4.2
- Accelerate version: 0.29.2
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: bf16
        - use_cpu: False
        - debug: False
        - num_processes: 8
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - enable_cpu_affinity: False
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- PyTorch version (GPU?): 2.1.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@amyeroberts

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

When using gradient_checkpointing, `CLIPEncoder` calls the `self._gradient_checkpointing_func`
https://github.com/huggingface/transformers/blob/51bcadc10a569847b93a30dbe3a077037ae63bad/src/transformers/models/clip/modeling_clip.py#L628
However, it seems that this attribute is defined in `PretrainedModel`, while `CLIPEncoder` is not a subclass of `PretrainedModel`.
This bug can be reproduced using the following code
```
from transformers import CLIPVisionModel
vision_tower = CLIPVisionModel.from_pretrained("openai/clip-vit-large-patch14-336")
print(vision_tower.vision_model.encoder._gradient_checkpointing_func)
```
which reports `'CLIPEncoder' object has no attribute '_gradient_checkpointing_func'`

### Expected behavior

Currently, I replace the `self.__gradient_checkpointing_func` directly with `torch.utils.checkpoint.checkpoint`, and it works when training with gradient checkpointing. However, I think this is not a perfect solution, since we need `__gradient_checkpointing_func` to handle the gradient_checkpointing_kwargs.

Comments:
Hi @TideDra, thanks for opening this issue! 

You first have to enable gradient checkpointing for the model:

```py
In [1]: from transformers import CLIPVisionModel
   ...: vision_tower = CLIPVisionModel.from_pretrained("openai/clip-vit-large-patch14-336")
   ...: vision_tower.gradient_checkpointing_enable({"use_reentrant": True})
   ...: vision_tower.vision_model.encoder._gradient_checkpointing_func
Out[1]: functools.partial(<function checkpoint at 0x15accd240>, use_reentrant=True)
```

Thanks for your help!
Reporting a vulnerability
Author: immi19
Pull request: no

@Michellehbn  @huggingface
need access to the Huggingface Private Bug Bounty Program

https://hackerone.com/hugging_face

I request the team to send me an invite for the Huggingface Private Bug Bounty Program

My Hackerone Id: sheikhimmi

I had already email to the security team but not yet get any response 

Hope the team will send me an invite soon

Regards

Comments:
cc @Michellehbn 

Done!
Add OLMo family
Author: 2015aroras
Pull request: no
Labels: name: New model

### Model description

OLMo is a series of Open Language Models designed to enable the science of language models. The OLMo models are trained on the [Dolma](https://huggingface.co/datasets/allenai/dolma) dataset. OLMo is releasing all code, checkpoints, logs (coming soon), and details involved in training these models.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Authored by Allen Institute for AI (HF org: [allenai](https://huggingface.co/allenai))

Implementation: https://github.com/allenai/OLMo/tree/main/olmo
HF Hub: [OLMo-1B](https://huggingface.co/allenai/OLMo-1B), [OLMo-7B](https://huggingface.co/allenai/OLMo-7B), [OLMo-7B-Twin-2T](https://huggingface.co/allenai/OLMo-7B-Twin-2T)

Weights in HF formats (`.safetensors` and `.bin`) can be found in the respective HF Hub page. Each of these repos has branches containing intermediate checkpoints.
Weights in the original OLMo format can be retrieved following the instructions on the Github page.

Comments:
Skipping Layers in a forward pass
Author: IsNoobgrammer
Pull request: no

### Feature request

How can we make input tokens pass through only selected layers/blocks 
for ex:- In Mistral every input tokens pass through ```embedding -> then 32 layers -> lmhead```

But how can we make that Input tokens pass through only ```embedding -> first 20 layers and last 6 layers only -> lmhead```


### Motivation

Various Reasearch Shows that there are layers redunduncy in Transformers architechture
Especially last layers of models are redundant

### Your contribution

-

Comments:
Hi @IsNoobgrammer, thanks for raising an issue! 

This is a more of a question for our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for specific feature requests and bug reports.

If you want to use fewer layers, you could just hack it like this: 

```py
from transformers import AutoModelForCausalLM

checkpoint = "mistralai/Mistral-7B-Instruct-v0.2"
model = AutoModelForCausalLM.from_pretrained(checkpoint)

model.model.layers = model.model.layers[:20] + model.model.layers[-6:]
```
Fix quality Olmo + SDPA
Author: fxmarty
Pull request: yes

Byproduct of me not merging/rebasing after https://github.com/huggingface/transformers/pull/29890, conflicting with https://github.com/huggingface/transformers/pull/30070

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30302). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

For the record some olmo slows tests were already failign before this PR (running on A100). Otherwise all good.

```
FAILED tests/models/olmo/test_modeling_olmo.py::OlmoIntegrationTest::test_model_1b_logits - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index ...
FAILED tests/models/olmo/test_modeling_olmo.py::OlmoIntegrationTest::test_model_7b_greedy_generation - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index ...
FAILED tests/models/olmo/test_modeling_olmo.py::OlmoIntegrationTest::test_model_7b_logits - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index ...
FAILED tests/models/olmo/test_modeling_olmo.py::OlmoIntegrationTest::test_model_7b_twin_2t_logits - RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu! (when checking argument for argument index ...
```
transformers version problem
Author: Oten-arch
Pull request: no

### System Info

Linux, transformer==4.38.2, python=3.10

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. When starting the DBGPT project (Baichuan Model，The official default installation for the DBGPT project is the latest version of transformer) and installing transformer>4.34.0, an error will be reported as AttributeError: 'BaichuanTokenizer' object has no attribute 'sp_model‘

2. However, if Transformer>4.34.0 is not installed, an error message will appear: ImportError: cannot import name 'is nltk available' from 'Transformers. utils. import utilslib/Python 3.10/site packages/Transformers/utils/import_utils. py)

3. It seems that both attributes are present when there is no transformer version. How can we solve this problem

### Expected behavior

I want to start the DBGPT project correctly without reporting the above two errors, and find a suitable transformer version

Comments:
pip install transformers==4.36.0 still reports AttributeError: 'BaichuanTokenizer' object has no attribute 'sp_model'

Hi @Oten-arch, thanks for raising an issue! 

I'm not aware of the DBGPT project, could you link to it, as well as providing a minimal code snippet which reproduces the issue? 


> Hi @Oten-arch, thanks for raising an issue!
> 
> I'm not aware of the DBGPT project, could you link to it, as well as providing a minimal code snippet which reproduces the issue?

DBGPT Github project address: https://github.com/eosphoros-ai/DB-GPT
DBGPT deployment installation document address: https://www.yuque.com/eosphoros/dbgpt-docs/urh3fcx8tu0s9xmb (I deployed step by step according to the source code)

> pip install transformers==4.36.0 still reports AttributeError: 'BaichuanTokenizer' object has no attribute 'sp_model'

me to,have you solved it

@Oten-arch You appear to be replying to yourself? 

The issue is arising in the DBGPT repo, and so should be reported there. 

`sp_model` was added a few releases ago to our tokenizers to fix a bunch issues we had with inconsistent behaviour if the tokenizer used SentencePiece. Any tokenizer which inherits from PreTrainedTokenizer should make sure that all of the appropriate attributes set. You can find an example here: https://github.com/huggingface/transformers/blob/3b6e95ec7fb08ad9bef4890bcc6969d68cc70ddb/src/transformers/models/albert/tokenization_albert.py

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add Flash Attention 2 to M2M100 model
Author: visheratin
Pull request: yes

# What does this PR do?

This PR adds support for Flash Attention 2 in M2M100 models (e.g., NLLB). [Here](https://colab.research.google.com/drive/1QaNgDMuUYAa6T_F5IrsfQzAwBA9ZwFyQ?usp=sharing) is the Colab notebook with a working demo.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@ArthurZucker @younesbelkada

Comments:
The `_use_flash_attention_2` is needed to handle different attention mask formats. As far as I can see, the same flags/logic is used across many other models (e.g., [DistilBERT](https://github.com/visheratin/transformers/blob/main/src/transformers/models/distilbert/modeling_distilbert.py#L702)). Is there another better way to handle this?

@visheratin correct, for llama it's because the attention mask logic has been refactored in favor of `AttentionMaskConverter`. OK to use `_use_flash_attention_2`, the other points are still valid though, can you check what has been done for GPT2 and make the changes accordingly? 🙏 

I fixed inheritance and added the sections about FA2 along with speedup image to the NLLB and M2M100 doc pages. I also added an integration test. Let me know if there is anything that needs to be done.

Sure! I committed the change.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30256). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

My pleasure! Thank you both, @amyeroberts and @younesbelkada, for the fast review!
RagRetriever fails to find faiss-gpu installed with pip not conda
Author: Berowne
Pull request: no

- `transformers` version: 4.5.1
- Platform: Linux-4.14.231-173.361.amzn2.x86_64-x86_64-with-glibc2.9
- Python version: 3.6.10
- PyTorch version (GPU?): 1.8.1+cu102 (True)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: yes nVidia V100 16Gb
- Using distributed or parallel set-up in script?: just single task in //
- rag: @patrickvonplaten, @lhoestq


## Information
Model I am using (RAG Retriever ...):

The problem arises when using:
[*  ] the official example scripts: (worked!)
[* ] my own modified scripts: (give details below)

The tasks I am working on is:
[*  ] my own task or dataset: (give details below)

## To reproduce 
Steps to reproduce the behavior:

1. Run transformers/examples/research_projects/rag/use_own_knowledge_dataset.py
This step worked fine yesterday prior to reboot.
2. Try to inspect output dataset directly using RagRetriever model in python... 3.6 > 
```python
from transformers import RagRetriever, RagSequenceForGeneration, RagTokenizer
retriever = RagRetriever.from_pretrained('facebook/dpr-ctx_encoder-single-nq-base', cache_dir=cache_dir, index_name="custom", indexed_dataset='./rag/out')
ImportError: 
RagRetriever requires the faiss library but it was not found in your environment. Checkout the instructions on the
installation page of its repo: https://github.com/facebookresearch/faiss/blob/master/INSTALL.md and follow the ones
that match your environment.
```

Also, if you import faiss, then faiss.__version__ does not exist.


Note for our environment we have to pip install faiss-gpu rather than conda since conda repos are blocked at proxy.
qds/NLP/aws_nlp/rag/out

A sample script to query the /path/to/my_knowledge_dataset/ would be handy.

Comments:
Got it to work with rebuild... and pip install faiss and faiss-gpu
git clone https://...github.../rag
export TOKENIZERS_PARALLELISM=false
pip install torch torchvision  ray[default] datasets faiss faiss-gpu matplotlib seaborn pandas transformers awscli s3fs scikit-plot
python use_own_knowledge_dataset.py --csv_path ./text.csv --output_dir ./out/text


@Divyansh-Gemini This looks more like a pip issue, as [faiss is available on pypi](https://pypi.org/project/faiss/). I'd suggest upgrading pip and trying again:

```
pip install --upgrade pip
pip install faiss
```

Thanks @amyeroberts for your response.
The issue has been resolved now.
I was installing `faiss`, but `faiss-cpu` was needed to be installed.