noamaanMulla-03/transformers-issues huggingface dataset:
Compile Grounding DINO
Author: pspdada
Pull request: no
Labels: name: Feature request

name: Vision

name: Compilation

name: Multimodal

### Feature request

I found that the Gounding DINO model `IDEA-Research/grounding-dino-base` cann't be compiled.
When I use `torch.comple(<The model>)`, it raises many error, such as `TypeError: unhashable type: 'dict'`.
can this be implemented?

### Motivation

It's very slow for Grounding DINO to perform batch inference, so I want some way to speed it up.

### Your contribution

Not sure.

Comments:
maybe cc @yonigozlan who is working on compile for with a few vision models

Yes! I'll get to it once I have a bit more bandwidth, but in the meantime if you want to have a go at it, part of the solution should be similar to what was done in https://github.com/huggingface/transformers/pull/33600
NotImplementedError: data_seed requires Accelerate version `accelerate` >= 1.1.0. This is not supported and we recommend you to update your version.
Author: zwhe99
Pull request: no
Labels: name: bug

### System Info

transformers==4.46.0

### Who can help?

@MekkC

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import TrainingArguments
TrainingArguments(output_dir=".", data_seed=42)
``` 

### Expected behavior

The latest version of Accelerate is 1.0.1 for now.

Comments:
According to [this comment in the fix PR](https://github.com/huggingface/transformers/pull/33731#issuecomment-2402756789) it seems there's a corresponding PR in the accelerate repo huggingface/accelerate#3150 that's needed to enable this, and which didn't make it into accelerate 1.0.1. Is accelerate 1.1.0 planned to release soon?

cc @muellerzr, do you know what's happening here?

Yes accelerate will be released by Friday w/ this capability, for the time being please install accelerate from git:
`pip install git+https://github.com/huggingface/accelerate`

Thanks, I got it!

> accelerate                        1.1.0.dev0 transformers                      4.47.0.dev0

> NotImplementedError: data_seed requires Accelerate version `accelerate` >= 1.1.0. This is not supported and we recommend you to update your version.
Refactor Roberta using modular
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

Refactor Roberta using **modular**. It also improves the modular converter in the following ways:

- If any decorators are used on top of classes/methods in the `modular_xxx.py`, those are used instead of the decorators in the original class
- If any variables used in the decorators are missing, automatically look for them in the visited files
- Automatically removes unused variables that may be written as part of the decorators dependencies, but that are not used if the decorators were overwritten
- Fix the `replace_call_to_super` which was not behaving correctly if any statements appeared before the `super()` in the `modular_xxx.py`


All modified files come from the copied from Roberta.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33923). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @ArthurZucker, could you review the changes please? 🤗

Ok, we now have a better dependency mapping when replacing class nodes in `ModularConverterTransformer`:
- we only look for (recursive) dependencies of the **new node** (after the call to `replace_call_to_super`). This avoids adding wrong dependencies, and thus removes the need for post-cleaning of classes/functions
- This is important to not rely on post-cleaning for functions/classes because since imports will be taken care of in the post-cleaning, we could otherwise still keep some of them wrongly (if a "to-be-removed" function/class is the only reason to keep an import)
- if we have no inheritance from a model-specific class, only check the files for functions imports, and add them and their dependencies if needed

IMO the logic in `leave_Class` in `ModularConverterTransformer` is much simpler now, let me know what you think I may be biased 😅🤗
Useful Sensors Moonshine Transcription Model
Author: njeffrie
Pull request: no
Labels: name: New model

### Model description

Model description can be found in the [Moonshine Whitepaper](https://github.com/usefulsensors/moonshine/blob/main/moonshine_paper.pdf).

I will be porting our [existing torch model](https://github.com/usefulsensors/moonshine/blob/b2a61fff243dd78ee2fa72dd1bceff8ccf656c4c/model.py) to Transformers.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

[Implementation](https://github.com/usefulsensors/moonshine) Special credit to @keveman for training and @evmaki for data collection and preprocessing. 
[Model weights](https://huggingface.co/UsefulSensors/moonshine)

Comments:
Hi @njeffrie, generally our recommendation is to port models as "custom code" models first. In these models, the code is included in the model repo itself, rather than in the core `transformers` library. You can write custom code models yourself without needing to open a PR to `transformers`, and users can still load them with `AutoModel.from_pretrained()`. Take a look at [the docs](https://huggingface.co/docs/transformers/en/custom_models) here for more information.

Excellent, thanks for the pointer! I'll definitely do this first, then we will consider creating a PR to add the model to `transformers` in the slightly longer term.

For now I'll keep this open for reference and for the future PR, unless you'd prefer I close then re-open the issue alongside a future PR to `transformers`.

I think you can close this for now, and reopen it later if you want to push for a full library PR!
Different results obtained using pipeline (worse) vs. model.generate under the same decoding strategy
Author: kirk86
Pull request: no
Labels: name: Core: Pipeline

name: bug

name: Generation

### System Info

4.44
Linux
3.12

### Who can help?

@Rocketknight1 
@gante 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

1. Train a small model like T5 on a small synthetic data for summarization
2. Evaluate the model on the test set
3. Results match validation accuracy from train logs when using `model.generate` with default `decoding_strategy = greedy search` when setting `do_sample=False, num_beams=1`
4. Try to replicate the same using the pipeline but results are completely off $\pm 50\\%$
5. Double checked all parameters to be set to default values as in `model.generate` still the same issue.
6. Iterated through all possible `decoding_strategies` and still no sign of correction when using `pipeline`

### Expected behavior

Expected the behaviour of pipeline to be the same as that of `model.generate` when using the same underlying `decoding_strategy`

Comments:
Hi @kirk86 -  could you upload your model (or reproduce the issue with an existing T5 model on the Hub), then share a short but complete reproducer script that shows the discrepancy? It'll make it a lot easier for us to track down any bugs!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix Input Handling Error in Floating Point Operations
Author: eugene123tw
Pull request: yes

# What does this PR do?

Fixes `OneFormerModel` has a list of inputs i.e  `main_input_name = ["pixel_values", "task_inputs"]` , `estimate_tokens` called in `floating_point_ops` can't process list of inputs thus raised an error:

```python
if self.main_input_name in input_dict:
Exception has occurred: TypeError       (note: full exception trace is shown but execution is paused at: _run_module_as_main)
unhashable type: 'list'
```


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Hi @eugene123tw, thanks for opening a PR! 

Could you provide a bit more context about the error being encountered, ideally with a code snippet to replicate?
Update setup.py
Author: NavikaAgarwal
Pull request: yes

---

## What does this PR do?

This PR updates the `setup.py` file, improving the structure for packaging, dependency management, and release processes. The changes include:

- Cleaning up any stale build artifacts during setup.
- Updating dependencies to the latest supported versions.
- Adding optional dependencies under specific groups like `torch`, `tensorflow`, and `onnx`.
- Including a custom command for updating the dependency version table, ensuring consistency across the package.
- Enhancing overall installation instructions and commands for better usability.
  
These improvements help in streamlining the build and release process, making it easier to maintain and contribute to the repository.

Fixes # (issue)

## Before submitting
- [x] This PR fixes dependency management and improves `setup.py`.
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? N/A (No prior discussion required for this enhancement).
- [x] Documentation has been updated to reflect the changes in the setup process.
- [x] New tests are not required as this change focuses on build scripts and packaging.

## Who can review?

Anyone familiar with packaging or dependency management can review this PR. Tagging relevant maintainers for specific areas:

- @muellerzr (HF Trainer/Accelerate)
- @SunMarc (Big Model Inference)
- @Rocketknight1 (pipelines, TensorFlow)

---

Comments:
I think this is maybe the most breaking PR I've ever seen to `transformers`, lol. I like your courage! 

For something this big, though, we strongly recommend:

1) Opening an issue first to discuss the problems you have with the current `setup.py` and get feedback
2) Keeping any PR as drafts, and making sure that tests pass locally / in the CI before pinging reviewers

PRs that touch this many things are likely to have lots and lots of side-effects, so we'll be _very_ wary about merging them unless there's a strong reason, and lots of evidence that we won't totally break builds / backward compatibility / etc.
Saving checkpoints *only* on improvement
Author: umarbutler
Pull request: no
Labels: name: Feature request

### Feature request

When using the Hugging Face Trainer, I would like to save a checkpoint only if my objective metric has improved.

### Motivation

Currently, I am using eval_steps=100,save_steps=100, save_limit=1 and load_best_model_at_end=True which means that every 100 steps, the latest checkpoint is getting written and then the previous checkpoint is getting deleted unless it is the best checkpoint.

This has done approximately 2TB of wear to my SSD in only a few days due to an excessive amount of checkpointing. I really don’t need to resume from the latest checkpoint, I just need the best checkpoint to be saved, and I’m not concerned about the run crashing, so in this case, there is really no need to be saving every 100 steps.

Additionally, it is not feasible to wait until the end of the run and load the best state because I am manually early stopping my runs. I do not wish to automate the early stopping either.

I’m happy to monkey patch my build of transformers if anyone is aware of the culprit lines I can comment out or modify.

### Your contribution

N/A

Comments:
Maybe you can use `save_strategy` argument to save best checkpoint.
https://github.com/huggingface/transformers/blob/33868a057c02f0368ba63bd1edb746be38fe3d90/src/transformers/trainer.py#L3014-L3015
How to retrain the GLIP model on the Object365 dataset
Author: Polarisamoon
Pull request: no

Since I made some modifications to the GLIP model, I need to perform some pre-training again to improve performance. I replaced `_base_ = [../_base_/datasets/coco_detection.py]` with `_base_ = [../_base_/datasets/objects365v1_detection.py]` in `glip_atss_swin-t_a_fpn_dyhead_16xb2_ms-2x_funtune_coco.py` to train on Object365. Is this correct?

Comments:
Yes, that modification is correct if you want to pre-train GLIP on Objects365 instead of COCO. However, make sure the Objects365v1 dataset config file exists and is properly configured.
get_input_embeddings return None in SpeechT5Model, which will make testcase failure
Author: sywangyi
Pull request: no
Labels: name: bug

name: Audio

### System Info

- `transformers` version: 4.45.0.dev0
- Platform: Linux-4.18.0-425.3.1.el8.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.0
- Huggingface_hub version: 0.24.5
- Safetensors version: 0.4.2
- Accelerate version: 0.30.1
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: NO
        - mixed_precision: no
        - use_cpu: False
        - debug: False
        - num_processes: 1
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: 6
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - enable_cpu_affinity: False
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): 2.15.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA A100 80GB PCIe



### Who can help?

@amyeroberts @ylacombe 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

see https://github.com/huggingface/transformers/pull/33194 which adds test test_peft_gradient_checkpointing_enable_disable.


pytest tests/models/speecht5/test_modeling_speecht5.py -k "peft"

FAILED tests/models/speecht5/test_modeling_speecht5.py::SpeechT5ModelTest::test_peft_gradient_checkpointing_enable_disable - AttributeError: 'NoneType' object has no attribute 'register_forward_hook'
FAILED tests/models/speecht5/test_modeling_speecht5.py::SpeechT5ForSpeechToSpeechTest::test_peft_gradient_checkpointing_enable_disable - AttributeError: 'NoneType' object has no attribute 'register_forward_hook'



### Expected behavior

test pass.

Comments:
Hey @sywangyi, thanks for opening this issue!
Would you be interesting to solve this and open a PR in that case?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
'model.config.max_length' is set to 20 in Hugging Face 'transformers' Whisper implementation but 448 in OpenAI Whisper
Author: AbdoKhaled-2021
Pull request: no
Labels: name: bug

name: Generation

name: Audio

### System Info

**Environment:**
- OS: [Ubuntu 22.04.5 ]
- Python version: [Python 3.10.14]
- Hugging Face `transformers` version: [4.41.0]
- Whisper model: `openai/whisper-large-v3-turbo`

### Who can help?

@sanchit-gandhi 
@ylacombe 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

**Steps to Reproduce:**

1. In Hugging Face `transformers`:
   ```python
   from transformers import WhisperForConditionalGeneration, WhisperProcessor

   model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-large-v3-turbo")
   processor = WhisperProcessor.from_pretrained("openai/whisper-large-v3-turbo")

   # Check model config max_length
   print(model.config.max_length)  # Outputs: 20
   ```

2. In OpenAI Whisper:
   ```python
   import whisper

   model = whisper.load_model("turbo")

   # Check relevant dimension
   print(model.dims)  # Outputs: n_text_ctx = 448
   ```

### Expected behavior

**Expected Behavior:**
The `max_length` for the Hugging Face Whisper model should ideally align with the OpenAI Whisper default, which has a text context length of 448. Alternatively, there should be a clear explanation or documentation regarding why the `max_length` differs, and users should be advised to adjust it for longer transcriptions.


Comments:
Hey @AbdoKhaled-2021, if you simply do `print(model.generation_config.max_length)`, you'll see that the max length is actually 448 for this checkpoint!

The implementation will use `generation_config.max_length` rather than `config.max_length`, so the behavior is as expected.

Hope it helps



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
How to build and evaluate a vanilla transformer?
Author: Bachstelze
Pull request: no

### Model description

"Attention Is All You Need" is a landmark 2017 research paper authored by eight scientists working at Google, responsible for expanding 2014 attention mechanisms proposed by Bahdanau et al. into a new deep learning architecture known as the transformer with an encoder, cross-attention, and a decoder.

### Open source status

- [X] The model implementation is available
- [ ] The model weights are available

### Provide useful links for the implementation

EncoderDecoderModels are supported via the huggingface API. Though it isn't possible to evaluate them properly: https://github.com/huggingface/transformers/issues/28721
How is it possible to build and evaluate a vanilla transformer with an encoder, cross-attention, and a decoder in huggingface?

Comments:
Hi @Bachstelze, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

> Though it isn't possible to evaluate them properly: https://github.com/huggingface/transformers/issues/28721

This isn't quite right - it's not possible to load them through the `AutoModelForCausalLM` API and hence submit to the open LLM leaderboard. It can [still be done manually](https://github.com/huggingface/transformers/issues/28721#issuecomment-2109920000). If the decoder is loaded with `AutoModelForCausalLM`, which is done by default, you already have the task specific head. To evaluate the model, you then just need inputs, labels and a metric.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
VLM: special multimodal Tokenizer
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Part of Major VLM standardization (https://github.com/huggingface/transformers/issues/33948). We will have special tokens that are present in all VLMs to be part if `XXXTokenizer` attributes. This will make our lives easier when doing several processing manipulations and/or formatting the prompt manually, as we can simply call `self.tokenizer.image_token`. 

Currently if we need any of VLM special tokens, those are saved in processor config, but not all models save it since not all models use it when calling the processor. After this PR I'll go over models and clean up the processing code given the changes. But we might still have to support old way, because we can't change stuff if that can break loading configs from the hub

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34461). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Should be ready for review @ArthurZucker ! 

I think we'll support simple non-multimodal tokenizers for quite a while in VLMs, no idea yet how/when to make this a new default