noamaanMulla-03/transformers-issues huggingface dataset:
Sync VQA pipeline with huggingface_hub spec
Author: Rocketknight1
Pull request: yes


Comments:
This PR will require the matching change here: https://github.com/huggingface/huggingface.js/pull/978

The cause is that the `huggingface.js` pipeline has a `label` output, which `transformers` pipelines do not.

This PR should now be ready, but is waiting on a release of huggingface_hub with the new spec from https://github.com/huggingface/huggingface.js/pull/978 before we can merge it

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34290). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@LysandreJik sure! The simplest approach is probably just to allow some arguments to mismatch the spec, especially arguments like `timeout` that just control some specifics of pipeline behaviour, rather than actually changing the output. Do you want me to go back and un-deprecate it?

I think _it's fine_ if `transformers` implements extra arguments that we don't expect inference users to use themselves :+1: 

In that case, maybe I'll:

1) Un-deprecate `timeout` (which is a little awkward, since we just released a version with the deprecations, but not that many people use it so it's probably no big deal)
2) Modify the spec tests to have a list of permitted arguments that can be missing in the spec without causing the test to fail?

WDYT @Wauplin?

Sounds like a good plan yes! And let's keep in mind that this list of extra arguments should really be reserved for "parameters that are not meant to be used with inference".

Yes sounds great

Re-added the `timeout` arg here - we'll need to merge the PR permitting that arg and rebase this one before the CI will pass.
[Object detection pipeline] Lower threshold
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR proposes to lower the default threshold of the object detection pipeline. It is currently set to 0.9, which causes a lot of inference widgets to not show any detected objects.



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30710). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

FYI, we usually use much more lower score threshold (e.g. 0.3) to see just tendency of the model. (+ Note that transformers has transformer architecture which have lower confidence level)

https://github.com/facebookresearch/detr/issues/216

https://github.com/IDEA-Research/DINO/blob/main/inference_and_visualization.ipynb

@NielsRogge should we merge it, or do you have something to add?

I haven't tried the pipeline yet with various thresholds. Do you think we should lower it even more? It could result in too many detections. Wondering what the best default value is

@NielsRogge We can always lower it further if necessary. I'd merge as-is and based on observations / feedback we can always update. 

With my experiments on cppe-5 dataset `threshold=0.4..0.5` visually looks fine, otherwise, too many FP appear.

With a quick search in other frameworks, I found for some models:
 - In Utlralitics - 0.25 / 0.3
 - In Mmedection - 0.05 / 0.3 / 0.5

But in general, it depends on the model and dataset

Ok thanks, will merge in that case.
[bugfix] incorrect usage of warning_once
Author: youkaichao
Pull request: yes

Hi, I'm a developer for the vllm library. We use transformers a lot!

Recently, we find that this simple code will emit many lines of error:

```python
from vllm.transformers_utils.config import try_get_generation_config
try_get_generation_config("BAAI/bge-multilingual-gemma2", False)
``` 

The error log cannot be suppressed by `try-except`, leading to something like:

```text
TypeError: not all arguments converted during string formatting
Call stack:
  File "/data/youkaichao/vllm/testf.py", line 2, in <module>
    try_get_generation_config("BAAI/bge-multilingual-gemma2", False)
  File "/data/youkaichao/vllm/vllm/transformers_utils/config.py", line 403, in try_get_generation_config
    return GenerationConfig.from_model_config(config)
  File "/data/youkaichao/miniconda/envs/vllm/lib/python3.9/site-packages/transformers/generation/configuration_utils.py", line 1235, in from_model_config
    generation_config = cls.from_dict(config_dict, return_unused_kwargs=False, _from_model_config=True)
  File "/data/youkaichao/miniconda/envs/vllm/lib/python3.9/site-packages/transformers/generation/configuration_utils.py", line 1093, in from_dict
    config = cls(**{**config_dict, **kwargs})
  File "/data/youkaichao/miniconda/envs/vllm/lib/python3.9/site-packages/transformers/generation/configuration_utils.py", line 475, in __init__
    self.validate(is_init=True)
  File "/data/youkaichao/miniconda/envs/vllm/lib/python3.9/site-packages/transformers/generation/configuration_utils.py", line 751, in validate
    logger.warning_once(
  File "/data/youkaichao/miniconda/envs/vllm/lib/python3.9/site-packages/transformers/utils/logging.py", line 328, in warning_once
    self.warning(*args, **kwargs)
Message: 'You have set `use_cache` to `False`, but cache_implementation is set to hybrid. cache_implementation will have no effect.'
Arguments: (<class 'UserWarning'>,)
```

I think this is caused by incorrect usage of `warning_once` . I have searched for all usages of `warning_once` , it should not have that `UserWarning` argument.

Hope it helps.

Comments:
cc @gante
Qwen2VL: skip base `input_ids`-`inputs_embeds` equivalence check
Author: gante
Pull request: yes

# What does this PR do?

Computing `inputs_embeds` in `Qwen2VL` is more complex than simply embedding `input_ids` (see `Qwen2VLForConditionalGeneration`), so the basic check doesn't apply :)

Fixes:
```
py.test tests/models/qwen2_vl/test_modeling_qwen2_vl.py::Qwen2VLModelTest::test_generate_from_inputs_embeds_1_beam_search --flake-finder --flake-runs 100
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34535). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
avoid calling `gc.collect` and `cuda.empty_cache`
Author: ydshieh
Pull request: yes

# What does this PR do?

Let's avoid calling `gc.collect` and `cuda.empty_cache` while the tests are running on CPU:
- those operations are slow
- (actually, in most cases, they are only relevant for integration tests which use large models)

Running on GPT2 tests,

60 seconds on main, 20 seconds on this PR

Comments:
Yes, a helper method is nice. Will update

updated. 

So far it doesn't call `gc.collect()` at all (default value `False`).
I would like to see if this would cause issue.
In general, we don't need to call it after each test method (so in `tearDown`) as it is slow.




The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34514). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
feat: add benchmarks pg indexes
Author: McPatate
Pull request: yes
Labels: name: run-benchmark

We want to make queries efficient as the content of the benchmarks tables grows, making dashboards more responsive and reducing load on the database. Hence the addition of indexes on the most used fields for filtering/joining.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34536). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
make `test_eager_matches_sdpa_inference `less flaky
Author: ydshieh
Pull request: yes

# What does this PR do?

With `torch.bfloat16` the numerical difference/instability occurs quite often, especially with multiple hidden layers.

This PR first changes `test_eager_matches_sdpa_inference` to create models with only 1 hidden layer. 

**number of failures per 500 runs**
|   | main | n_layer=1 |
| ------------- | -------------: | -------------: |
| llama  | 16  | 2  |
| idefics2   | 391  | 15  |

Then it relaxes the condition a bit: only checks 80% of the sequences. If the results match on those 80%, the test pass.

This makes the test much less flaky. On 500 runs, it pass (for llama, mistral, idefics2 and Llava)

Finally, change the image size of `llava` and `VipLlava` from `30` to `8` so the sequence length is much smaller and avoid numerical issues.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34512). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Bug Fix for issue #34294
Author: fpgaminer
Pull request: yes

Update SiglipVisionEmbeddings.forward to cast input to correct dtype before embedding it.

Fixes #34294


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@amyeroberts @qubvel


Comments:
[CLIPSeg] Make interpolate_pos_encoding default to True
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR fixes #34415 by making `interpolate_pos_encoding` default to `True`.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34419). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Please see my comment https://github.com/huggingface/transformers/issues/34415#issuecomment-2438972495 before merging.

I've updated my PR description based on your comments

I just reverted to the interpolation which is used by all other models, so the logits diff remains (edit: it looks like it has to do with my local setup)
Support batched input for decoder start ids
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

This PR addresses [issue #28763 ](https://github.com/huggingface/transformers/issues/28763). The requested feature already work out of the box, I just made it explicit and added one line in the docs. 

The changes were tested with `pytest -k generate_input tests/generation/test_utils.py`

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).


## Who can review?
@gante 

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28887). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts or @ArthurZucker Hi, PR is ready to review

@amyeroberts it should be ready to merge -- with a `list` as input (as opposed to a tensor), there are no serialization issues 🤗 

@amyeroberts can we merge this PR? The failing test is unrelated (`test_save_load_fast_init_from_base`, i.e. model init), and I've opened [this PR](https://github.com/huggingface/transformers/pull/28930) to tag it as flaky. Its flakiness is discussed on our internal slack [here](https://huggingface.slack.com/archives/C01NE71C4F7/p1707407250079089).

@gante Yep! I can merge
Blip: get/set input embeddings correctly
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/34109 and adds `get_input_embeddings` method to the retrieval model. Also fixes the same methods in BLIP model where we should be working with text embeddings. Returning vision embeddings will not be able to resize the vocab size

Added tests as those were all skipped and thus we never knew there was an issue

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34152). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Also :  we might need / want to force return_dict to TRUE, to avoid all the if else

would make it simpler!
BLIP: enable generation tests
Author: zucchini-nlp
Pull request: yes

# What does this PR do?


Enables generation tests for BLIP models, except BLIP-1 (turned out to be a bit harder). I changed the generation tests to use `modelTest.input_name` as BLIP is the only model that uses pixel values as main input and thus checking generated text length's will always fail. 

I tried to get rid of custom generate for these models, but that opened a Pandora box so I think better not waste time on an old model and maintain it for a while, until the model gets deprecated. But still I did some changes so we don't need to add extra `bos` at the beginning and now the decoder-based BLIP models return full text at output. Encoder-decoder based models return only generated text, which is consistent with what an LLM should do

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34174). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

>  changed the generation tests to use modelTest.input_name as BLIP is the only model that uses pixel values as main input and thus checking generated text length's will always fail.

I'd like very much to avoid this change -- extra logic for all tests to handle a niche corner case. Let's brainstorm alternatives! `main_input` in the generative tests is used to check the shapes. Perhaps we always want to look for `input_ids` or `inputs_embeds` in the input dictionary? 🤔 

@gante I tried to force `input_ids` always and I found another corner case with ~~Whisper which expects `input_features` to be the main input for shape checking~~. I would very much love to make BLIP standard and maybe I'll make so in v5, because it will break a whole lot of things.

OMG, I found an option while writing this reply, whisper and the other audio model are encoder-decoder so we can make it work by getting main input in decoder-only models. Just before the check happens, in the same indent block  :)
If you agree, I'll make the change hehe


> OMG, I found an option while writing this reply, whisper and the other audio model are encoder-decoder so we can make it work by getting main input in decoder-only models. Just before the check happens, in the same indent block :)
If you agree, I'll make the change hehe

if it works, sounds good! (make sure to leave a comment) 

@gante requesting re-review, since the `input-name` was merged as a separate PR. I rebased `main` and ran tests again
ValueError during training with streaming dataset.
Author: mxjmtxrm
Pull request: no
Labels: name: trainer

name: bug

name: data

### System Info

- `transformers` version: 4.44.0
- Platform: Linux-5.15.0-88-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.24.7
- Safetensors version: 0.4.5
- Accelerate version: 0.34.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.0a0+81ea7a4 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA H800

### Who can help?

@muellerzr @SunMarc

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I want to do training with streaming dataset, as my dataset is super large. The code like the following:
```
dataset = load_dataset('data_path', streaming=True)
dataloader = DataCollatorForLanguageModeling(tokenizer,mlm=False)

trainer = Trainer(
        model=model,
        tokenizer=tokenizer,
        args=training_args,
        train_dataset=train_dataset if training_args.do_train else None,
        eval_dataset=valid_dataset if training_args.do_eval else None,
        data_collator=dataloader,
    )

```

I met the following error:
```
File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 611, in __init__
    raise ValueError(
ValueError: The train_dataset does not implement __len__, max_steps has to be specified. The number of steps needs to be known in advance for the learning rate scheduler.
```
How to solve this problem? or is there another way to train with large datasets?

### Expected behavior

-

Comments:
Hello! Have you followed this example to use with iterable datasets? https://huggingface.co/docs/datasets/v2.14.5/en/stream#stream-in-a-training-loop

cc @lhoestq 

As the error suggest you should specify max_steps. It is required to know how many steps your training should do for the learning rate, but since we often can't know the size of a dataset in advance you should specify max_steps manually.

Since I stripped hf code of all the valueerrors my skin has cleared up and my sleep has improved.

If you remove that `ValueError` @sine2pi you'll get a much more arcanic error message :)
What's the issue with this error? It seems pretty indicative of an issue to me, and contains the code to solve it

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Xmod model has not module 'roberta'
Author: WLucky
Pull request: no
Labels: name: Usage

name: bug

### System Info

latest version

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

path:  src/transformers/models/xmod/modeling_xmod.py

function `def freeze_embeddings_and_language_adapters(self)`

when i call this function, the error rasie:  **model no module roberta**
a similar error raise when load my funtinue xmod checkpoint

and i modify as this (delete roberta),  it seem run ok, but I cant certain its really right:

```
def freeze_embeddings_and_language_adapters(self):
        """
        Freeze the embeddings and language adapters of the model. Usually, this is applied before the model is
        fine-tuned on a downstream task.
        """
        logger.info("Freezing embeddings")
        for parameter in self.embeddings.parameters():
            parameter.requires_grad = False
        logger.info("Freezing adapters")
        for layer in self.encoder.layer:
            if layer.output.adapter_layer_norm is not None:
                for parameter in layer.output.adapter_layer_norm.parameters():
                    parameter.requires_grad = False
            for parameter in layer.output.adapter_modules.parameters():
                parameter.requires_grad = False
```

### Expected behavior

call `freeze_embeddings_and_language_adapters ` with no error raise

Comments:
Hey @WLucky, do you mind sharing the code snippet that got you this error? As well as the full error, if you can

Thanks!

> Hey @WLucky, do you mind sharing the code snippet that got you this error? As well as the full error, if you can
> 
> Thanks!

The reproduce is simple and as follows:

```
model = XmodModel.from_pretrained(....)
model.set_default_language("en_XX")
model.freeze_embeddings_and_language_adapters()
```
Then the error will raise in last line. such as not find 'roberta'.

I cant reproduce the error as I already modify the code.
I suspect that XmodModel has no roberta module.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
:red_circle: :red_circle:  fix `query_pre_attn_scalar` different of `num_heads` in default gemma2 config
Author: molbap
Pull request: yes

# What does this PR do?

as per title. Previous default config was a nonworking default config, new defaults are aligned with gemma2-2b-it.

@ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34540). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Qwen2_moe: Avoid zero tokens fowarding for some experts
Author: Coco58323
Pull request: no
Labels: name: bug

### System Info

transformers=4.43.3
python=3.8
Linux

### Who can help?

@ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Under Auto-GPTQ with Triton kernel, it would use math.log2() function in Line 96 
<img width="874" alt="image" src="https://github.com/user-attachments/assets/ad1e299c-ba90-47c1-9926-028b417bf4cb">
While for the Implementation of MoE, 

https://github.com/huggingface/transformers/blob/f7396876849926afa87c9412d67c43618dad403d/src/transformers/models/qwen2_moe/modeling_qwen2_moe.py#L667-L675

some experts directly forward with zero tokens and therefore the input shape is like [0, seq_length, hidden_states], and fails on log2()

The issue could be solved by checking the number of tokens before Line 675
`
if current_state.shape[0] == 0:
continue
`


### Expected behavior

No more forwarding with zero tokens.

Comments:
WOuld you like to open a PR for a fix? #31173 could be a nice addition as well? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

No completed, but we need to make sure it is worth it as this might break torch fx export!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.