noamaanMulla-03/transformers-issues huggingface dataset:
is_vision_availble() is slow and called a lot
Author: colossoi
Pull request: no
Labels: name: Feature request

name: Vision

### Feature request

Memoize the result of `is_vision_availble()`, because it depends on installed packages which are unlikely to change.

### Motivation

`is_vision_availble()` in import_utils.py is slow, in that it results in many calls to os.stat and os.list_dir, because it is checking import metadata.  Because it is used in places like `is_valid_image()`, it can be called A LOT when trying to bootstrap an image dataset.  For example in my use case of opening a PIL Image, and immediately transforming it with a transforms.Compose(), the calls to stat and list_dir account for more than half of the runtime.

### Your contribution

Sorry, I'm not a great python developer.  

Comments:
@collosi can you share a minimal code snippet to reproduce your issue?

@collosi happy to take this on once you send elaborate on how to reproduce the bug

A simple reproducer:

```
from transformers import AutoProcessor, Blip2ForConditionalGeneration
import requests
from PIL import Image
import timeit

url = 'https://avatars.githubusercontent.com/u/25720743?s=48&v=4'
image = Image.open(requests.get(url, stream=True).raw).convert('RGB')   
processor = AutoProcessor.from_pretrained("Salesforce/blip2-opt-2.7b")

def test():
    processor(image, return_tensors="pt")

print(timeit.timeit("test()", setup="from __main__ import test", number=5000))
```

Note that for each image processing, `is_vision_available` is called 63 times, directly or indirectly. I've pushed a PR (#29280) to cache this. In the above example, for 5000 images, we get to see a improvement from 13s->6s.
Use `DS_DISABLE_NINJA=1` during deepspeed build
Author: ydshieh
Pull request: yes

# What does this PR do?

Deepspeed ` 0.13.3` was released on Feb. 24 but causes the prebuild failed in our CI workflow.

@pacman100 found it's likely due to  https://github.com/microsoft/DeepSpeed/pull/5088

and suggested using 

> DS_DISABLE_NINJA=1

I tried it and the prebuild is fine now.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29289). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Use DS_DISABLE_NINJA=1
Author: ydshieh
Pull request: yes

# What does this PR do?

Deepspeed ` 0.13.3` was released on Feb. 24 but causes the prebuild failed in our CI workflow.

@pacman100 found it's likely due to  https://github.com/microsoft/DeepSpeed/pull/5088

and suggested using 

> DS_DISABLE_NINJA=1

I tried it and the prebuild is fine now.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29290). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Can not execute example in idefics-9b-instruct
Author: ppsmk388
Pull request: no

### System Info

tokenizers-0.15.1 
transformers-4.37.0 
python3.8.10
Linux

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

When I run example in https://huggingface.co/HuggingFaceM4/idefics-9b-instruct

```
import torch
from transformers import IdeficsForVisionText2Text, AutoProcessor

device = "cuda" if torch.cuda.is_available() else "cpu"

checkpoint = "HuggingFaceM4/idefics-9b"
model = IdeficsForVisionText2Text.from_pretrained(checkpoint, torch_dtype=torch.bfloat16).to(device)
processor = AutoProcessor.from_pretrained(checkpoint)

# We feed to the model an arbitrary sequence of text strings and images. Images can be either URLs or PIL Images.
prompts = [
    [
        "https://upload.wikimedia.org/wikipedia/commons/8/86/Id%C3%A9fix.JPG",
        "In this picture from Asterix and Obelix, we can see"
    ],
]

# --batched mode
inputs = processor(prompts, return_tensors="pt").to(device)
# --single sample mode
# inputs = processor(prompts[0], return_tensors="pt").to(device)

# Generation args
bad_words_ids = processor.tokenizer(["<image>", "<fake_token_around_image>"], add_special_tokens=False).input_ids

generated_ids = model.generate(**inputs, bad_words_ids=bad_words_ids, max_length=100)
generated_text = processor.batch_decode(generated_ids, skip_special_tokens=True)
for i, t in enumerate(generated_text):
    print(f"{i}:\n{t}\n")
```

I got:
```
 OSError: We couldn't connect to 'https://huggingface.co' to load this file, couldn't find it in the cached files and it looks like HuggingFaceM4/idefics-9b-instruct is not the path to a directory containing a file named processor_config.json.
Checkout your internet connection or see how to run the library in offline mode at 'https://huggingface.co/docs/transformers/installation#offline-mode'
```
in 
```
`processor = AutoProcessor.from_pretrained(checkpoint)
```

### Expected behavior

Successful code execution

Comments:
Hi @ppsmk388, thanks for raising an issue! 

I'm able to load the processor without issue. As the error indicates, this might be due to a connection issue. Could you re-run and try again? 

Thank you, it is a network problem

@ppsmk388 Thanks for confirming! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
altclip can not be traced by fx?
Author: TXacs
Pull request: no

### System Info

- `transformers` version: 4.37.1
- Platform: Linux-5.4.0-47-generic-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.3.1
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.2+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from config import load_config

from transformers import (
    AltCLIPModel,
    AltCLIPConfig,
)
from transformers.utils.fx import symbolic_trace

def main():
    config_kwarg = load_config("altclip")
    config = AltCLIPConfig(
            **config_kwarg
        )

    # create model
    model = AltCLIPModel(config)

    traced_model = symbolic_trace(model)

if __name__ == "__main__":
    main()
```

### Expected behavior

When I run the above code, it reports an error: `ValueError: You have to specify pixel_values.`

Somebody can help me？

Comments:
Hi @TXacs, thanks for raising this issue! 

You need to pass in `input_names` to `symbolic_trace`:

```py
from config import load_config

from transformers import (
    AltCLIPModel,
    AltCLIPConfig,
)
from transformers.utils.fx import symbolic_trace

def main():
    config_kwarg = load_config("altclip")
    config = AltCLIPConfig(
            **config_kwarg
        )

    # create model
    model = AltCLIPModel(config)

    traced_model = symbolic_trace(model, ['input_ids', 'pixel_values'])

if __name__ == "__main__":
    main()
```

> Hi @TXacs, thanks for raising this issue!
> 
> You need to pass in `input_names` to `symbolic_trace`:
> 
> ```python
> from config import load_config
> 
> from transformers import (
>     AltCLIPModel,
>     AltCLIPConfig,
> )
> from transformers.utils.fx import symbolic_trace
> 
> def main():
>     config_kwarg = load_config("altclip")
>     config = AltCLIPConfig(
>             **config_kwarg
>         )
> 
>     # create model
>     model = AltCLIPModel(config)
> 
>     traced_model = symbolic_trace(model, ['input_ids', 'pixel_values'])
> 
> if __name__ == "__main__":
>     main()
> ```

Thanks a lot! It solved.
Add `non_device_test` pytest mark to filter out non-device tests
Author: fxmarty
Pull request: yes

As per title.

Many tests in transformers do not use `torch_device`, `.to(0)`, `.to("cuda")`, etc. and are always run on CPU. These tests are irrelevant for people who would like to run Transformers tests on their device.

With this proposed PR, one can now run `pytest tests/ -m "not not_device_test" --collect-only` to filter out non-device tests. We use [`pytest_collection_modifyitems`](https://docs.pytest.org/en/7.1.x/reference/reference.html#pytest.hookspec.pytest_collection_modifyitems) in order to both capture the tests defined in `test_modeling_common.py` and in subclasses as well (no need to `@pytest.mark.not_gpu_test` everywhere) which does not add a significant overhead after tests collection.

We prefer a mark `non_device_test` rather than `device_test`, as people adding new tests may forget to add the device tests in the list. It is safer to be greedy (in a reasonable extent).

This now gives

```
================================================= 34914/54481 tests collected (19567 deselected) in 10.58s ==================================================
```

instead of the previous 54481 tests selected.

This is requested by AMD.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29213). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Actually now getting `=========== 27353/54481 tests collected (27128 deselected) in 9.77s ============` skipping all the pipelinemixin tests that are run always on CPU.
Add feature extraction mapping for automatic metadata update
Author: merveenoyan
Pull request: yes

This PR automatically adds vision models that can be used with the new feature extraction pipeline to a mapping so that their pipeline tag will change. 
Note: I only added vision models _and_ the models in which image encoder architecture exists separately (e.g. saw Siglip Vision Model class so I added it)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28944). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I have checked that there were changes to change black to ruff, so I made sure I had the right version of ruff, but make style and make quality doesn't seem to make any changes for some reason.

I made the variable a public one yet it's not importable from what I see in CI.

@merveenoyan In order to be able to import the objects, it's necessary to add them to the `auto` and `transformers` init. 

Essentially, any place you find `"MODEL_MAPPING"` in the inits, you'll need to add the equivalent for `"MODEL_FOR_IMAGE_MAPPING"`:
* E.g. [here](https://github.com/huggingface/transformers/blob/3f4e79d29ce32d9f8f75b082836b01ee180d0966/src/transformers/models/auto/__init__.py#L42) and [here](https://github.com/huggingface/transformers/blob/3f4e79d29ce32d9f8f75b082836b01ee180d0966/src/transformers/models/auto/__init__.py#L226) in `src/transformers/models/auto/__init__.py`
* [here](https://github.com/huggingface/transformers/blob/3f4e79d29ce32d9f8f75b082836b01ee180d0966/src/transformers/__init__.py#L6181) and [here](https://github.com/huggingface/transformers/blob/3f4e79d29ce32d9f8f75b082836b01ee180d0966/src/transformers/__init__.py#L1450) for `src/transformers/__init__.py` 

You'll also need to add the required `_LazyAutoMapping` class in `modeling_auto.py` e.g. [here](https://github.com/huggingface/transformers/blob/3f4e79d29ce32d9f8f75b082836b01ee180d0966/src/transformers/models/auto/modeling_auto.py#L1206C17-L1206C33)

That being said - this means this object is exposed at the top level of `transformers`, which I think is OK. Another alternative would be to have a list of objects to skip in the `check_repo.py` script.

@amyeroberts I think CI was broken from another PR and I merged main into this branch now, it seems to work. Can you review this? 

@amyeroberts would you like to merge?
RWKV: raise informative exception when attempting to manipulate `past_key_values`
Author: gante
Pull request: yes

# What does this PR do?

Some generation methods (like the new ngram speculation thingy) need to manipulate `past_key_values`. RWKV, a recurrent neural network, doesn't have this attribute -- a standard `AttributeError` is raised when such methods are called with RWKV. (related comment: https://github.com/huggingface/transformers/pull/27775#issuecomment-1897404295)

This PR improves the error message, explaining what's happening and what to do.

NOTE: some newer RWKV variants use custom modeling code, so this PR won't affect them. I'll point the users to this PR if the issue pops up.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28600). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix ROPE embeddings for LLama
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

This [test](https://app.circleci.com/pipelines/github/huggingface/transformers/84847/workflows/2a5e5769-9431-4e2b-babb-81a112558a97/jobs/1098065) failed on my PR and I checked to see the reason. I found that the changes introduced to make llama compile compatible are causing the issue. 

The fixes here are tested with fullgraph compile, compilation is still working without graph breaks. Additionally, the failing [test]((https://app.circleci.com/pipelines/github/huggingface/transformers/84847/workflows/2a5e5769-9431-4e2b-babb-81a112558a97/jobs/1098065)) was run 500 times. I found that aside from rope embeddings, the cause of test failure was in SDPA attention. I cannot say what is the reason exactly, but running the test 500 times gives 95% success in SDPA and 100% success in eager, using the fixes introduced in this PR.  Prior to these fixes, the tests were running with 90% success for both attentions.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@gante 


Comments:
Fix flaky test vision encoder-decoder generate
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes #28841. I could not exactly reproduce the failing test, but the issue seems to be the `EOS` stopping generation earlier than it reaches max_length. 

Before adding the fix I checked that `EOS` was indeed indicated in the generation_config as not `None`.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@gante 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28923). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix max_length criteria when using inputs_embeds
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes #28953 . StoppingCriteria with max_length behaves differently when provided `input_ids` or `inputs_embeds`, this happens only on decoder-only models. The PR fixes it so that the criteria accounts for the length of `input_embeds` when generating


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@gante 

Comments:
oh, i see, added a new fix and checked that creating an empty tensor does not break anything

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28994). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts unrelated CI failures, I believe this can be merged 🤗 

@zucchini-nlp Can you try rebasing? Fixes should have been merged into main with resolve the currently failing tests 

@amyeroberts thanks, now it's all green and can be merged
Use `torch.bool` instead of `torch.int64` for non-persistant causal mask buffer
Author: fxmarty
Pull request: yes

Adding `self.register_buffer("causal_mask", torch.triu(causal_mask, diagonal=1), persistent=False)` in @ArthurZucker's rewrite of llama & gemma adds a 500 MB overhead when serializing to ONNX/TorchScript IR/PyTorch ExportedProgram (from https://pytorch.org/docs/stable/export.html), for `max_position_embeddings=8182`.

Essentially, these IRs do not support non-persistent buffers. One quick fix is to use torch.bool instead of torch.int64, but bool is still 8-bits in pytorch (https://github.com/pytorch/pytorch/issues/41571) & the overhead is still ~70 MB.

The lowered overhead is acceptable to me, but this won't scale to 10M context length.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29241). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Can confirm this shrunk my [tiny-random-GemmeForCausalLM](https://huggingface.co/Xenova/tiny-random-GemmaForCausalLM) ONNX export from ~500MB to ~70MB ([PR](https://huggingface.co/Xenova/tiny-random-GemmaForCausalLM/discussions/2)). Ideally, there would be no overhead, but I think this helps a ton for now!

@amyeroberts Running on A100, I can confirm that no additional tests are failing with `RUN_SLOW=1 CUDA_VISIBLE_DEVICES=2 pytest tests/ -k "llama or gemma" -s -vvvvv` compared to running on `main`.
🌐 [i18n-ZH] Translate chat_templating.md into Chinese
Author: shibing624
Pull request: yes

# What does this PR do?

Translate chat_templating.md into Chinese
part of https://github.com/huggingface/transformers/issues/20095


## Who can review?
Documentation: @stevhliu 
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag members/contributors who may be interested in your PR.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28790). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

 @yyLeaves @jiaqiw09  @bojone @yuanzhoulvpi please help to reviewing the translate work.
[i18n-vi] Translate README.md to Vietnamese
Author: hoangsvit
Pull request: yes

# What does this PR do?

This PR adds a Vietnamese translation for the README.md.

## Before submitting

- [ ]  This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Documentation: @stevhliu and @MKhalusova


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29229). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Nice job!
> 
> I can't read any Vietnamese so unfortunately, I can't review the content itself. Do you know any other Vietnamese-readers in the ML community who would be able to review your work?

Could you please help me, @davidnguyen-94 ?

> > Nice job!
> > I can't read any Vietnamese so unfortunately, I can't review the content itself. Do you know any other Vietnamese-readers in the ML community who would be able to review your work?
> 
> Could you please help me, @davidnguyen-94 ?

Sure, I'd be happy to help!
I have checked and the translation is completely reasonable. @stevhliu @hoangsvit
Converting TF2 SavedModel models to Huggingface
Author: jhyuklee
Pull request: no

Hi, I'd like to know how I can convert a TF2 SavedModel (e.g. [gtr-base-1](https://www.kaggle.com/models/google/gtr/frameworks/tensorFlow2/variations/gtr-base/versions/1?tfhub-redirect=true)) to a Huggingface PyTorch model as described in the [README](https://huggingface.co/sentence-transformers/gtr-t5-base). We have a similar model in the same format and would like to use it in Huggingface. Thanks!

Comments:
cc @Rocketknight1 

Hi @jhyuklee, take a look at the `convert-gtr` notebook in the HuggingFace sentence transformers repos [here](https://huggingface.co/sentence-transformers/gtr-t5-base/blob/main/convert-gtr.ipynb). This shows the process of converting a model from the TF Hub, so you should be able to adapt that to convert other SavedModels.

Hi @Rocketknight1, this is exactly what I was looking for. Will close this once I try that out.
Fixed Deformable Detr typo when loading cuda kernels for MSDA
Author: EduardoPach
Pull request: yes

# What does this PR do?

Fixes a typo that was overlooked at https://github.com/huggingface/transformers/pull/29133 when loading the cuda kernels for Deformable Detr

c.c. @amyeroberts 

Comments:
> Thanks for fixing!
> 
> Could you run the slow model tests on a GPU? Once we can confirm everything is passing we can merge 🤗

Ran them on Colab and they all passed 
![image](https://github.com/huggingface/transformers/assets/69953243/18304c8a-3d3f-43ad-96e1-8860ed3de5e3)