noamaanMulla-03/transformers-issues huggingface dataset:
(testing our CI)
Author: gante
Pull request: yes

# What does this PR do?

Nothing to see here :)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32915). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

(identified cause, closing PR)
Uniform kwargs for processors of audio-text models
Author: leloykun
Pull request: yes

# What does this PR do?

- Uniformizes kwargs for processors of audio-text models.
- An extension of https://github.com/huggingface/transformers/issues/31911
- NOTE: don't review nor merge until this PR is complete: https://github.com/huggingface/transformers/pull/32841

TODO Models:
- [x] Clap
- [x] CLVP
- [x] MusicGen Melody
- [ ] Qwen2 Audio
- [x] Seamless M4T
- [x] SpeechT5
- [x] Wav2Vec2 Bert

TODO tests
- [ ] Add audio-text-specific processor tests
- [ ] Remove unnecessary/duplicated tests

Models with special args (will not be done in this PR):
- PopPiano

Models with weird `_in_target_context_manager` logic (will not be done in this PR):
- MusicGen
- SpeechToText
- Wav2Vec2
- Wav2Vec2 w/ LM
- Whisper

Fixes # (issue)

- https://github.com/huggingface/transformers/issues/31911

## Who can review?

@zucchini-nlp @molbap @yonigozlan


Comments:
@zucchini-nlp the tests are failing because of this: https://github.com/huggingface/transformers/pull/32921
No module named 'transformers.modeling_flash_attention_utils'
Author: Leo-T-Zang
Pull request: no
Labels: name: bug

### System Info

transformers in /usr/local/lib/python3.10/dist-packages (4.42.4)

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. from transformers.modeling_flash_attention_utils import _flash_attention_forward

### Expected behavior

1 from transformers.modeling_flash_attention_utils import _flash_attention_forward

ModuleNotFoundError: No module named 'transformers.modeling_flash_attention_utils'

Comments:
Hi @Leo-T-Zang, thanks for opening an issue! 

The addition of this function into the modeling_flash_attention_utils module is not yet part of a pypi release. You can do this using an install from the development branch: `pip install git+https://github.com/huggingface/transformers`

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Update docker image building
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Properly update cc @gante

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32918). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

(@ArthurZucker shall we use your admin permissions to push this? The images are usually built at 2am Paris time, so the problem would be fixed tomorrow with no further manual inputs from us 🙌 )

Yeah no worries!
add scaling_factor to GemmaRotaryEmbedding for fix error in GemmaLine…
Author: nprotasov
Pull request: yes

In generated code, there are GemmaLinearScalingRotaryEmbedding and GemmaDynamicNTKScalingRotaryEmbedding which use scaling_factor need to declare it in base class, or remove generation of: GemmaLinearScalingRotaryEmbedding and GemmaDynamicNTKScalingRotaryEmbedding

Comments:
cc @ArthurZucker @gante 

@ArthurZucker I run: python utils/diff_model_converter.py --files_to_parse src/transformers/models/gemma/diff_gemma.py
But got error: FileNotFoundError: [Errno 2] No such file or directory: 'ruff'
What is ruff file?

This PR can be merged, but will quickly be superseded by https://github.com/huggingface/transformers/pull/31999 :D 

(@nprotasov in the PR I linked, Gemma will gain access to all RoPE scaling strategies :) )

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[GroundingDino] - GroundingDinoProcessor kwargs is Broken
Author: EduardoPach
Pull request: no

### System Info

While working on https://github.com/huggingface/transformers/pull/31828 I realized that `GroundingDinoProcessor.__call__` passes the kwargs only to the `self.tokenizer` which is not ideal specially if one wants to pass `annotations` to `self.image_processor` as it's currently not possible. 

It should be an easy fix following https://github.com/huggingface/transformers/pull/31197 

### Who can help?

@amyeroberts 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoProcessor
from datasets import load_dataset

processor = AutoPocessor.from_pretrained("IDEA-Research/grounding-dino-tiny")
ds = load_dataset("EduardoPacheco/aquarium-sample", split="train")
id2label = {0: "fish", 1: "jellyfish", 2: "penguins", 3: "sharks", 4: "puffins", 5: "stingrays", 6: "starfish"}
prompt = ". ".join(id2label.values()) + "."

# Passes `annotations` as kwargs to tokenizer instead of image_processor
inputs = processor(images=ds["image"], text=[prompt, prompt], annotations=ds["annotations"], return_tensors="pt")
```

### Expected behavior

Kwargs being correctly mapped to tokenizer and image processor

Comments:
We're tracking them all at https://github.com/huggingface/transformers/issues/31911, cc @zucchini-nlp I guess we can add Grounding DINO to that list

Sure, added to the list. Guess we can close this issue now and track Grounding DINO in 31911 :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Jamba: update integration tests
Author: gante
Pull request: yes
Labels: name: run-slow

# What does this PR do?

🟢 Fixes `generate`-related integration tests for jamba 🟢 

I've checked them against:
- my machine (compute capability 8); 
- our slow CI (compute capability 7) -- CI is still red, but due to a SDPA test.

⚠️ skips logits checks on older devices: there are big differences across different versions, possibly explained by custom cuda kernels. Given the relatively low usage of jamba, we don't have the bandwidth to dive.

_________________________

Detective work 🕵️ 
1. The tests use a [dummy model](https://huggingface.co/ai21labs/Jamba-tiny-random), `ai21labs/Jamba-tiny-random`, the generation text quality doesn't matter.
2. On my machine, checking out to the [commit](https://github.com/huggingface/transformers/pull/29943/commits/a3e8094af98690f149320a9cf3b7d9cc21dd50ea) that added the current tests, I already get a difference. My machine has the same major compute capability as an A100 (RTX4090).
3. We can see in the original PR that the tests were being updated as changes were made. There were a few commits after the latest version of the tests that modify the outputs, which would explain the failing tests since day 1

Comments:
cc @ydshieh 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32250). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

(thank you for trigger the tests on the runner 🙏 )
fix: Added missing `huggingface_hub` installation to workflows
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
The `utils/notification_service.py` file requires `huggingface_hub` package.
https://github.com/huggingface/transformers/blob/85345bb439652d3f03bb4e123cef7a440f2ba95b/utils/notification_service.py#L30

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@ydshieh @amyeroberts 

Comments:
Thank you @Sai-Suraj-27 . We should move to using `.github/workflows/slack-report.yml`, but at this moment, this PR makes sense.

> Thanks!
> 
> The failing tests should have been resolved with some updates to the CI images. I can't re-trigger the CI runs, as they failed too long ago. @Sai-Suraj-27 could you push an empty commit to try and retrigger a run?

Thanks for the review @amyeroberts. Done 👍🏻. All the tests are passing now.
Fix attention propagation for vision towers of llava-like models
Author: qubvel
Pull request: yes
Labels: name: Vision

name: run-slow

# What does this PR do?

`CLIP` vision model now [supports sdpa attention](https://github.com/huggingface/transformers/pull/31940) and automatically dispatch on sdpa if possible. Llava-like models utilize AutoModel.from_config to initialize vision tower, however, `attn_implementation` was not propagated, which caused errors in CI: the model was in eager mode, and at the same time CLIP vision tower was with sdpa attention.

cc @zucchini-nlp @ydshieh 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32221). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

 @zucchini-nlp If I get it right your concern is regarding the situation

  - vision model **doesn't** support sdpa
  - text model **do** support sdpa
  - and we pass `attn_implementation="sdpa"`
 
Am I right?

The question here is how should we specify attention implementation, should we dispatch it automatically or let the user make a choice? For example instead of passing `attn_implementation="sdpa"` make it `attn_implementation={"vision_config": "eager", "text_config": "sdpa"}`


@qubvel my idea to to dispatch automatically, as that is backwards compatible and I think most VLM-related models support SDPA already. We'll raise a warning if either one doesn't support SDPA, so that the user knows what's happening in the background.

I already have some code for that, so that we infer SDPA flag from config and try to dispatch to whichever that supports it. But that means we'll have to slightly change tests, because tests skip based on uninitialized classes sdpa flag, which will be False by default for all Llavas. Can make a PR soon and let's see if that works for you

Ok, that sounds good, let's wait until your PR is ready to see if we still need this fix!
`run_summarization` likely race condition creates flaky `test_tensorflow_examples`
Author: molbap
Pull request: no
Labels: name: Tests

As in title. See for instance https://app.circleci.com/pipelines/github/huggingface/transformers/100640/workflows/f4e73659-1f14-45da-8740-7314e788fb72/jobs/1340353 that happened in #32651 . Can't reproduce locally. 
Caused by https://github.com/huggingface/transformers/blob/9d2ab8824c2a820a0ac9f2cd884ef838ca77b481/examples/tensorflow/summarization/run_summarization.py#L62-L70 :

Will cause a failure at test collection

```bash
_______ ERROR collecting examples/tensorflow/test_tensorflow_examples.py _______
examples/tensorflow/test_tensorflow_examples.py:67: in <module>
    import run_summarization
examples/tensorflow/summarization/run_summarization.py:63: in <module>
    nltk.data.find("tokenizers/punkt")
/usr/local/lib/python3.10/site-packages/nltk/data.py:554: in find
    return find(modified_name, paths)
/usr/local/lib/python3.10/site-packages/nltk/data.py:541: in find
    return ZipFilePathPointer(p, zipentry)
/usr/local/lib/python3.10/site-packages/nltk/compat.py:36: in _decorator
    return init_func(*args, **kwargs)
/usr/local/lib/python3.10/site-packages/nltk/data.py:394: in __init__
    zipfile = OpenOnDemandZipFile(os.path.abspath(zipfile))
/usr/local/lib/python3.10/site-packages/nltk/compat.py:36: in _decorator
    return init_func(*args, **kwargs)
/usr/local/lib/python3.10/site-packages/nltk/data.py:943: in __init__
    zipfile.ZipFile.__init__(self, filename)
/usr/local/lib/python3.10/zipfile.py:1271: in __init__
    self._RealGetContents()
/usr/local/lib/python3.10/zipfile.py:1338: in _RealGetContents
    raise BadZipFile("File is not a zip file")
E   zipfile.BadZipFile: File is not a zip file
------------------------------- Captured stderr --------------------------------

```
Would be nice to have: 
- isolate the behaviour of the bad file creation
- flag the flaky part of the failing test accordingly.


Comments:
See [this convo](https://github.com/huggingface/transformers/pull/32659#discussion_r1718297174): NLTK doesn't support parallelism :(

^ I think that PR is a proper fix :D
CI: separate step to download nltk files
Author: gante
Pull request: yes

# What does this PR do?

Should fix #32667 : NLTK is complaining about parallelism. The parallelism stems from the `pytest` call, so we should get rid of the problem if we download the NLTK data in advance of the `pytest` call 🤗 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32935). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[BUG] Offline loading of non-safe tensors fails
Author: pseudotensor
Pull request: no
Labels: name: bug

### System Info

4.41.0 or others
python 3.10
Ubuntu 22


### Who can help?

@ArthurZucker and @younesbelkada 

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Online do:
```
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained('mistralai/Mistral-7B-Instruct-v0.2', use_safetensors=False)
```
Then offline do:
```
import os
os.environ['HF_HUB_OFFLINE'] = '1'
os.environ['TRANSFORMERS_OFFLINE'] = '1'

from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained('mistralai/Mistral-7B-Instruct-v0.2')
```
You'll see it fail with connection error despite setting offline envs:

```
Traceback (most recent call last):
  File "/home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/urllib3/connection.py", line 198, in _new_conn
    sock = connection.create_connection(
  File "/home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/urllib3/util/connection.py", line 60, in create_connection
    for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
  File "/home/jon/miniconda3/envs/h2ogpt/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution
```

For now, I patched the problem like this: https://github.com/h2oai/h2ogpt/blob/main/docs/trans.patch

With the patch all goes fine.

Thanks!

### Expected behavior

If setting offline envs and are offline, it shouldn't fail if have all files just because of the safe tensors checks in the code.

I imagine there may be many other offline issues like this that need fixing in some general way.

Comments:
Hi @pseudotensor !
I think fixing this makes sense, what about adding the new except branch here: https://github.com/huggingface/transformers/blob/c876d12127272ce2886ce51399928c3343d475b9/src/transformers/utils/hub.py#L649 instead of patching it on `modeling_utils` so that we fix that for all methods / code paths that use `has_file`. What do you think?

Probably fine, you would definitely know best.

Perfect, would you like to contribute for that ? Otherwise happy to do it !

Please do.
FIX / Hub: Also catch for `exceptions.ConnectionError`
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes: https://github.com/huggingface/transformers/issues/30920 

Please see the detailed issue description for more details, it seems in transformers we hit a corner case when using HF offline mode + loading a non-safetensors model offline 

cc @Wauplin as I am not 100% sure about this and why this is not catched within `hf_raise_for_status` in `huggingface_hub`



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31469). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@younesbelkada I'm surprised that a `requests.exceptions.ConnectionError` can be raised by `hf_raise_for_status`. In theory, `raise_for_status` (from Python's `requests` package) is meant for when a HTTP response has been returned by the server and you want to raise if the server returned an error. A ConnectionError is different and happens before or while getting a response from the server. So to catch it you should do it [here](https://github.com/huggingface/transformers/blob/71e3856904a2f6d6da407649b7935b3428f55fb0/src/transformers/utils/hub.py#L662) IMO.

While you are at it, you can also include read timeout and remove SSLError and ProxyError for which it's better to raise:

```py
        except (requests.exceptions.SSLError, requests.exceptions.ProxyError):
            # Actually raise for those subclasses of ConnectionError
            raise
        except (
            requests.exceptions.ConnectionError,
            requests.exceptions.Timeout,
            OfflineModeIsEnabled,
        ):
            return has_file_in_cache
``` 
This is what we do in `huggingface_hub` [download method](https://github.com/huggingface/huggingface_hub/blob/a626cc3dbcadbf83817c3c345a65510976399435/src/huggingface_hub/file_download.py#L744).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@younesbelkada @huggingface/transformers-core-maintainers I still think this PR is worth updating + merging (see previous comment https://github.com/huggingface/transformers/pull/31469#issuecomment-2192210035).

@Wauplin I updated following your comments, but not 100% I'm catching in the right way. LMK!

@Wauplin Thanks for writing the correct code! As we both contributed to this - let's get a final pair of eyes to double check this is OK
`eval_on_start` triggers `AttributeError` in JupyterLab
Author: matteosantama
Pull request: no
Labels: name: trainer

name: bug

### System Info

`transformers`: '4.44.0'
`Python`: '3.10.14 | packaged by conda-forge | (main, Mar 20 2024, 12:45:18) [GCC 12.3.0]'

### Who can help?

@muellerz @SunMarc 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Here is a self-contained script that triggers the error:

```python
model_id = "distilbert/distilbert-base-uncased"

model = AutoModelForSequenceClassification.from_pretrained(
    model_id, num_labels=2
)
tokenizer = AutoTokenizer.from_pretrained(model_id, use_fast=True)

ds = (
    Dataset.from_dict({"text": ["Hello world!"], "labels": [0]})
    .map(
        lambda b: tokenizer(b["text"], truncation=True),
        batched=True,
    )
)


trainer = Trainer(
    model=model,
    args=TrainingArguments(
        output_dir="temp",
        eval_strategy="steps",
        save_strategy="no",
        eval_on_start=True,  # <-- setting to False avoids the error
    ),
    tokenizer=tokenizer,
    train_dataset=ds,
    eval_dataset=ds,
    compute_metrics=lambda _: {"metric": 1.0},
)
trainer.train()
```

The error I get is

```
AttributeError: 'NotebookTrainingTracker' object has no attribute 'value'
```

which seems related to the fact that I'm running the code in a Jupyter notebook.

### Expected behavior

I should be able to use the `eval_on_start` parameter without triggering an error.

Comments:
Hi,

I have found that the cause of this exception is that in Jupyter Notebook, for a `transformers.Trainer`, when its argument `eval_on_start` is set to `True`, running `Trainer.train()` will lead to the following function execution order:
1. `NotebookProgressCallback.on_train_begin()`
2. `NotebookProgressCallback.on_prediction_step()`
3. `NotebookProgressCallback.on_train_end()`

Running `NotebookProgressCallback.on_prediction_step()` directly after `NotebookProgressCallback.on_train_begin()` will result in two attributes of `NotebookProgressBar`, i.e., `value` and `label`, being used before assigned (see methods `update()`, `update_bar()` and `display()` of `NotebookProgressBar` in [transformers/utils/notebook.py](https://github.com/huggingface/transformers/blob/60d5f8f9f04026cb801d0dc5158bf4531e250072/src/transformers/utils/notebook.py#L61) for details).

I have open a PR to fix this.
Gemma2: eager attention by default
Author: gante
Pull request: yes

# What does this PR do?

See title :)

We know that SDPA yields inferior modeling results, so we should use `eager` by default. This has been the source of some model quality GH issues, e.g. #32848 

Slow tests for gemma 2 ran, no regressions

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32865). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
IDEFICS2 integration test hits OOM on CI
Author: amyeroberts
Pull request: no
Labels: name: bug

### System Info

Running on main from commit 44f6fdd74f84744b159fa919474fd3108311a906 - c.f. CI runs for this PR: https://github.com/huggingface/transformers/pull/32275

Example run: https://github.com/huggingface/transformers/actions/runs/10134513823/job/28020840326 

Opening an issue to keep track

### Who can help?

@amyeroberts 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Push a commit with the message `[run_slow] idefics2` on a PR with the label `run-slow` to trigger

### Expected behavior

It doesn't run out of mem - the 4 bit integration test might be enough to test however

Comments: