noamaanMulla-03/transformers-issues huggingface dataset:
[Whisper] Bump threshold of FA2 equivalence tests
Author: sanchit-gandhi
Pull request: yes

# What does this PR do?

As reported in #29942, there are three Whisper FA2 tests that consistently fail on `main`. These tests are "flaky", in the sense that they'll pass 15 times out of 20.

Rather than marking them with the `flaky` decorator, which would re-trigger the tests a certain number of times, we increase the threshold by a factor of 2x. In a repeat of 20 tests, the tests pass all 20 times.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30861). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Cannot import name 'WhisperForAudioClassification -Already installed transformers==4.40.2
Author: manjualoshious
Pull request: no
Labels: name: Audio

ImportError: cannot import name 'WhisperForAudioClassification' from 'transformers' (/usr/local/lib/python3.10/dist-packages/transformers/__init__.py)

**from transformers import  WhisperForAudioClassification**

I have installed  **transformers==4.40.2**

Issue still exists.

Comments:
cc @sanchit-gandhi 

The issue has not been  resolved yet
Pls help

On Wed, 15 May, 2024, 9:19 pm amyeroberts, ***@***.***> wrote:

> cc @sanchit-gandhi <https://github.com/sanchit-gandhi>
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/30834#issuecomment-2112906071>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AN6MF7FWBEFG3PUGOK7E7JLZCN7ZZAVCNFSM6AAAAABHYNYQTCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMJSHEYDMMBXGE>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>


Hi @manjualoshious, please be patient refrain sending many comments on an issue and spamming people with notifications. We receive many pull requests and issues per day and they will be addressed in due time. Leaving a comment that something hasn't been addressed within an hour is not sustainable behaviour: we would be unable to meaningfully address any issues if everyone did this.

The best way to get a quick response is to follow the issue template and provide all of the information provided, which I've [already asked for you to do in another issue](https://github.com/huggingface/transformers/issues/30831#issuecomment-2112691307). A quick response is on the order of hours / days. 

Hey @manjualoshious - could you please provide the full traceback you're getting why you try this? It would also be great to see the output of `transformers-cli env`, so we can cross-check you have the appropriate packages installed. For reference, mine is:
```
- `transformers` version: 4.41.0.dev0
- Platform: macOS-14.4.1-arm64-arm-64bit
- Python version: 3.11.6
- Huggingface_hub version: 0.23.0
- Safetensors version: 0.4.2
- Accelerate version: 0.27.2
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.2.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): 0.8.1 (cpu)
- Jax version: 0.4.24
- JaxLib version: 0.4.24
```

And I am successfully able to import the class:
```python
from transformers import WhisperForAudioClassification
```

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
`SPMConverter` does not always add the user defined symbol -> slow fast is thus not equivalent
Author: ArthurZucker
Pull request: no


The `SPM` converter (and all that inherit / use it) does not take into account the user defined symbols, leading to issues like this one: https://huggingface.co/01-ai/Yi-9B/discussions/11#6643844f6ac3fe108e3d5190 



It also does not really take into account the prefix space param which can and should be extracted from the proto:
```python
split_by_unicode_script: true
split_by_number: true
split_by_whitespace: true
treat_whitespace_as_suffix: false
allow_whitespace_only_pieces: true
split_digits: true
```
and
```python3
normalizer_spec {
  name: "identity"
  precompiled_charsmap: ""
  add_dummy_prefix: false
  remove_extra_whitespaces: false
  normalization_rule_tsv: ""
}
``` 
cc @itazap, on a more general converter! 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
added resnet1d
Author: minesh1291
Pull request: yes

# What does this PR do?

Created ResNet model to work with 1d inputs

## Who can review?

Models:

- vision models: @amyeroberts

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
`AutoModel.from_pretrained(local_files_only=True)` changes the hash inside the `refs/main`
Author: kouroshHakha
Pull request: no

### System Info

transformers 4.38.1

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Download the model locally but put it under a hash that is not the same as the one from the latest in the hub
2. Run this. Interestingly if you instantiate the tokenizer first and the model it works fine. The culprit is that the `is_peft_available()` branch of the logic will look for `adapter_config.json` file and it won't find it, so it will ask hub and in that process it overrides the `refs/main` file. 


```
from transformers import AutoModel, AutoTokenizer
model = AutoModel.from_pretrained(
    model_id, 
    local_files_only=True,
)

tokenizer = AutoTokenizer.from_pretrained(
    model_id, 
    local_files_only=True,
)

```

### Expected behavior

The code above to work, more importantly `AutoModel.from_pretrained()` to not change the content of `refs/main` file.

Comments:
cc @younesbelkada and @Rocketknight1 

I totally lost track of this one - will take it now!

Hi @kouroshHakha, just to speed things up a bit, can you give me a specific model ID and commit hash to use in the reproduction script that triggers this bug?

I don't recall exactly. But if you take `mistralai/Mistral-7B-Instruct-v0.1` for example and then add the files under a different snapshot commit (both in the directory name and in the main file) then you can repro the issue. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Maybe also of interest to @Wauplin 

I'm still planning to investigate this when I get a chance too!

I feel that this is a misunderstanding of [how the cache-system works](https://huggingface.co/docs/huggingface_hub/guides/manage-cache#understand-caching). When downloading from `main`, it's normal that the `refs/main` file in the cache gets updated with the latest known commit for that ref. I'm not sure to understand what's the issue here. 

> then add the files under a different snapshot commit 

Just for context, files in the cache dir (e.g. in `.../snapshots/` should not be modified manually. It is supposed to be a read-only folder otherwise the cache system is corrupted.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Update git templates
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Update who to ping

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31539). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

:wave: 

Hope we'll continue to see your contributions @younesbelkada ;- )
Fix the error caused by incorrect use of logger in pipeline
Author: lanyun1103
Pull request: yes

# What does this PR do?

Fix the error caused by incorrect use of logger in pipeline. Before I found this problem, a friend also raised this problem here.

https://github.com/huggingface/transformers/issues/27678#issuecomment-2181363511

This commit is to solve this problem.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?

Comments:
Kosmos-2.5 implementation in transformers
Author: Natyren
Pull request: no
Labels: name: New model

name: Multimodal

### Model description

Hello everyone,

The Kosmos-2.5 is a multimodal literate model that can be used for tasks such as OCR and text-rich image comprehension. It includes a ViT encoder, a Resampler, and a shared decoder module. To the best of my knowledge, the architecture of this model is similar to Kosmos-2 but has some differences. Due to these differences, using this model in Transformers requires a standalone implementation.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Paper: https://arxiv.org/pdf/2309.11419
Code: https://github.com/microsoft/unilm/tree/master/kosmos-2.5
Authors: @Dod-o @wolfshow

Comments:
cc @ydshieh 

I would like to assist with this implementation. If there are any guidelines on how to do it effectively, I would like to join implementation process


Hi @Natyren, there's a guide in the documentation here: https://huggingface.co/docs/transformers/add_new_model

Hi @Natyren, sorry for the late reply. I am thinking to talk to the model author about if they are interested in porting this model into `transformers`. I will come back to you here for the updates.

Hi @ydshieh, I'm an intern at MSRA, and my mentor @Dod-o want to convert the ks25 to hf form. Is thre anything I can do for you.

Hi @tic-top Thank you for this message! This is a great news. It's probably better for me to add you into one of our slack channel. Let me check.

(just sent an email :-) now)

I would love this! What’s needed to move it forward?

@EwoutH We are collaborating with @tic-top to port this into `transformers` 🤗
Add Jinja as a requirement with the right version cutoff
Author: Rocketknight1
Pull request: yes

We (I) were lazy and didn't add Jinja as a requirement because it was already a requirement of `torch`. This PR finally makes it a requirement of `transformers` as well, and lets us set the right version cutoff so we can ensure we have the filters we need to render templates properly.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31536). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Merging this to make sure it's in time for the release, but @ydshieh feel free to let me know if there's anything else I need to add and I'll put it in a follow-up PR!

@Rocketknight1 Is `jinja` necessary in some tests? Or it's just for some people including you when working with templates?

@ydshieh there are chat template tests in [test_tokenization_common](https://github.com/huggingface/transformers/blob/main/tests/test_tokenization_common.py) and [test_chat_template_utils](https://github.com/huggingface/transformers/blob/main/tests/utils/test_chat_template_utils.py)

In order to have this version being taking into account in our CI, we will have to this dependency being used in 

- for example, `docker/torch-light.dockerfile` so CircleCI could use it
- also some docker files for GitHub Actions (daily CI for example)

If so far CI works well without what I suggest above, it's fine. We can take some actions once we need some changes.
Fix doc typo in `TrainingArguments`
Author: qgallouedec
Pull request: yes

# What does this PR do?

Minor typo fix
 
@stevhliu and @MKhalusova


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31503). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Transformers master version breaks compatibility with `torch<2.3`
Author: WeichenXu123
Pull request: no
Labels: name: Should Fix

### System Info

transformers master version: `pip install git+https://github.com/huggingface/transformers`
torch version < 2.3, e.g. `pip install "torch==2.2"`
python 3.9
Ubuntu 22.04

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers.utils.import_utils import is_torch_xpu_available

is_torch_xpu_available(check_device=True)
```
raises error `AttributeError: module 'torch' has no attribute 'xpu'`

```
>>> from transformers.utils.import_utils import is_torch_xpu_available
>>> is_torch_xpu_available(check_device=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/weichen.xu/miniconda3/envs/mlflow/lib/python3.9/site-packages/transformers/utils/import_utils.py", line 765, in is_torch_xpu_available
    _ = torch.xpu.device_count()
  File "/home/weichen.xu/miniconda3/envs/mlflow/lib/python3.9/site-packages/torch/__init__.py", line 1932, in __getattr__
    raise AttributeError(f"module '{__name__}' has no attribute '{name}'")
AttributeError: module 'torch' has no attribute 'xpu'
```

### Expected behavior

No error raised but return `False`

Comments:
Hi @WeichenXu123, thanks for reporting! 

I've opened #31573 which should fix this
Fix is_torch_xpu_available for torch < 2.3
Author: amyeroberts
Pull request: yes

# What does this PR do?

A recent PR #31238 updated the `is_torch_xpu_available` check to reflect XPU support from Pytorch >= 2.4. However, this broke backwards compatibility with versions of pytorch < 2.4 when `is_ipex_available` is `False` and `check_device=True`. 

Technically, this fails if the installed torch in the environment is <= 2.2, and 2.3 will still pass, as torch 2.3 has the `torch.xpu` module. However, as the intended logic is to default to the pytorch available logic for >= 2.4 and previously this evaluated to False for 2.3, this is more backwards compatible. 

Fixes #31563

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31573). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Added version constraint on numpy for version <2.0
Author: Resteklicken
Pull request: yes

# What does this PR do?
This PR fixes the numpy version in `setup.py` to versions below 2.0. At the moment only a lower bound for versions >= 1.17 is specified. The latest major release of numpy that dropped last week causes a bunch of issues with transformers and numpy should be constrained to version 1.xx until API compatibility is restored.

Fixes #31568 

@Narsil 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31569). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix typing errors in `Qwen2ForTokenClassification`
Author: kevinhu
Pull request: yes


Comments:
@kevinhu Running `make fix-copies` and pushing the changes should resolve the CI failures 

@amyeroberts Thanks for the tip, however running `make fix-copies` had the opposite effect of removing the proposed changes to Qwen2 rather than propagating them to the other files.

I've gone ahead and updated them so the tests pass now.
Siglip: add `_no_split_module`
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes #31400. Adds a '_no_split_module' in SigLipModel by combining those specified in Text and Vision components. 

Otherwise, when we call SiglipModel with `device_map` an error will be thrown because Text/Vision components are not children of SiglipModel and we need to add `_no_split_module`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31566). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
SiqlipVisionModel does not support "device map= auto": no split modules`attribute
Author: lucasjinreal
Pull request: no
Labels: name: Feature request

### Feature request

How to make SiglipVisionModel can support Auto map to map to multiple GPUs.

### Motivation

Currently, using MLLM with siglip, the whole model might need automap, since the vision encoder part is SiglipVisionModel, if it doesn;t support , would be a big problme

Also, would consider add flashattention support for Siglip?

### Your contribution

None for now

Comments:
Hi @lucasjinreal, thanks for opening a feature request! 

Could you share a code snippet of how the model is being created with `auto_map` and the running environment (run `transformers-cli env` in the terminal and copy-paste the output)? SigLip should support `device_map="auto"`

I checked with the code in [model-doc](https://huggingface.co/docs/transformers/en/model_doc/llava) for SigLip and also got an error. Supporting "device_map" in VLMs is indeed needed important. I believe `_no_split_modules` should be same as in [ClipModel](https://github.com/huggingface/transformers/blob/main/src/transformers/models/clip/modeling_clip.py)

For `flashattention` afaik current VLMs in transformers use optimized attn implementations only for LLM backbone (e.g. LLaVa supports Flash-Attn and SDPA even though CLIP doesn't). There's an issue for adding SDPA attn (#30565) to all VLMs, I can open another tracker-issue for Flash-Attn but will not able to work on it right now. Open for community contributions 

I have surpassed this error, by simply add a _no_split_modules = [] to the attribute.

But it could be better add inside transoformers, it's just a single line. I could submit a PR for this.

As for flashattn, it's a really needs, it can boost vlms training more faster.

@lucasjinreal cool, PR would be nice but you need to test in multi-gpu setting that everything is being split correctly. I don't think that an empty "split_modules" will work as the most similar CLIP doesn't split at some modules. If you don't have multiple gpus, I can run some tests after the PR is open :)

Flash-Attn noted, thanks, will add to my todo list!

I have been using empty list and make it can be deivicemap auto on multiple GPUs, currently inference is normal. I still didn't know why CLIPVIsionModel should make CLIPENcoderLayer didn't automap though.

@lucasjinreal i just noticed that SigLip already has `_no_split_modules` in TextModel and in VisionModel, yet not in the SigLipModel. If I do `_no_split_modules=[]` as you tried, device mismatch error is raised so we have to add text and vision models' `_no_split_modules` to enable it

LMK if you're up to opening a PR :)

Hi, In my cased I just using SIglipVisionModel as a parent class and used a SiglipVisionModelSplit(SiglipVisionModel) in my MLLM.

So I think it not appliable to inside of transformers.  Let me think a better way to do this



I believe the best solution is to copy 'no-split-modules' that are already indicated in text-vision components, and add them in SiglipModel's 'no-split-modules'
Something wrong for `StoppingCriteria`
Author: iopwsy
Pull request: no

### System Info

transformers==4.41.2
python==3.10

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The code is:
```python
class StopforGPT2(StoppingCriteria):
    def __call__(self, input_ids: torch.LongTensor, scores: torch.FloatTensor, **kwargs) -> bool:
        return input_ids[0,-1].detach().cpu().numpy() == 50256
model.generate(input_id.to(device),
                            do_sample=True,
                            top_k = 10,
                            top_p = 0.95,
                            temperature = 0.5,
                            max_new_tokens = 100,
                            stopping_criteria=StoppingCriteriaList([StopforGPT2()]))
```
It is well for `transformers==4.33.2` but wrong for `transformers==4.41.2`


The bug is `"bitwise_or_cuda" not implemented for 'Float'`

### Expected behavior

How to fix it

Comments:
`transformers==4.38.0` is OK but `transformers==4.39.0` exists bug

cc @gante 

Hey! 

StoppingCriteria was modified in #29116 to return a tensor of shape (batch_size,) where each entry is a boolean indicating if to stop or not. So you can do something like:

```python
eos_token = torch.tensor(50256, device=input_ids.device)
return torch.isin(input_ids[:, -1], eos_token)
```


Thanks very much

Thanks very much