noamaanMulla-03/transformers-issues huggingface dataset:
BART generate with min_new_tokens exceeds maximum length 
Author: vsocrates
Pull request: no
Labels: name: Generation

### System Info

- `transformers` version: 4.40.2
- Platform: Linux-4.18.0-477.36.1.el8_8.x86_64-x86_64-with-glibc2.28
- Python version: 3.10.14
- Huggingface_hub version: 0.23.0
- Safetensors version: 0.4.3
- Accelerate version: 0.30.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.3.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker @younesbelkada @gante

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

If I load a fine-tuned `BARTforConditionalGeneration` model and then try to generate text with it, I run into the following error: `This is a friendly reminder - the current text generation call will exceed the model's predefined maximum length (1024). Depending on the model, you may observe exceptions, performance degradation, or nothing at all.`

Generation code:
```python
outputs = model.generate(input_ids, attention_mask=attention_mask, num_beams=3, 
                         min_new_tokens=1500,
                         max_new_tokens=2500,
                         # stopping_criteria=stopping_criteria,
                         early_stopping=True)
```

I was under the impression that since the BART decoder generates autoregressively, there was no limit to its generation? 

### Expected behavior

Generation without a CUDA or out-of-bounds error with arbitrary length. 

Comments:
Hi @vsocrates 
Thanks for the issue ! 
You are getting that warning because the model's maximum positional embedding stops at 1024 tokens, some models have fixed positional embeddings where you can't exceed the maximum number of tokens by design (e.g. by having a `nn.Embedding` layer), for some models it is possible to exceed that, at your own risk as the model has not been trained to exceed that many tokens. If you are getting consistent / nice generations, I would say that there is nothing to worry about, otherwise you might need to use other models that support longer context length

Understood! Went through some other [issues](https://stackoverflow.com/a/77774322/2471385) and and it looks like T5 might use relative position embeddings so in theory, should be able to extend beyond its max context length (512 tokens), but potentially with some loss of accuracy/weird generations, is that correct? 

Yes this is correct, from my experience with flan-T5 that was possible but with some potential loss of accuracy / unconsistent generation

Great, thanks, closing this issue!
Add PaliGemma
Author: molbap
Pull request: yes

# What does this PR do?
This PR adds support for PaliGemma, a new VLM from Google.

## Who can review?
 @ArthurZucker


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30814). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Uploading own MusicGen checkpoint
Author: ElizavetaSedova
Pull request: no
Labels: name: Good Second Issue

### Feature request

Hi, I'm interested in the MusicGen model. I saw that you provide a conversion option for pre-trained models. But how can I load my own model? I don't see the possibility of using my own checkpoint. I carried out fine tuning from the original small model. I would be very happy to be given such an opportunity.

### Motivation

When I want to load my checkpoint using MusicgenForConditionalGeneration.from_pretrained(), I can't specify my path to it.

### Your contribution

I'm just wondering if there is a way or recipe to use my custom checkpoint.
src/transformers/models/musicgen/convert_musicgen_transformers.py

Comments:
Hey! you should be able to convert it using the script, `src/transformers/models/musicgen/convert_musicgen_transformers.py` did that not work for you? 

@ArthurZucker  I'm afraid this only works for pre-trained models that are publicly available. But not for a custom checkpoint. I checked the available arguments in convert_musicgen_transformers.py and saw that I couldn't specify my folder for "--checkpoint". I can only choose from ['small', 'medium', 'large'] or ['facebook/musicgen-stereo-small', 'facebook/musicgen-stereo-medium', 'facebook/musicgen -stereo-large']

In fact, this was resolved quite easily when I explicitly specified decoder_config = decoder_config_from_checkpoint("small") to get the configuration. But it would be better to have flexibility in this.

Would you like to open a PR to include this? Would be welcome! 🤗 

hi @ArthurZucker im intereseted in contributing to this issue, may i know how to replicate the issue.

Thanks!

I also noticed that the inference speed increased by 30% compared to the default inference provided by the musicgen developers with the same parameters, models and input data. I would be very interested to know how you managed to speed up the inference?

I don't remember it's been a while 😅 But I am pretty sure the original codebase is sequential, while we support batching

@ShamithaReddy you don't necessarily have to reproduce the issue, it's more like we need to add flexibility to the conversion script!
Image Classification Guide Missing Dependency and Addition for Local Models
Author: Tanman2001
Pull request: no
Labels: name: Good First Documentation Issue

### System Info

- `transformers` version: 4.39.2
- Platform: Windows-10-10.0.22631-SP0
- Python version: 3.11.1
- Huggingface_hub version: 0.22.1
- Safetensors version: 0.4.2
- Accelerate version: 0.28.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.2+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Tested using PyTorch only.

- Missing pip install scikit-learn

1. Follow Image Classification guide here: https://huggingface.co/docs/transformers/tasks/image_classification
2. Warning on `accuracy = evaluate.load("accuracy")` saying scikit-learn is missing.
3. Later training crashes due to missing module.

Additional recommendation:

Users should be able to use their model for inference even when they are unable to or it is not preferable for the user to push their model to the hub.

Add to guide under inference, in the pipelining section:
`classifier = pipeline("image-classification", model="<path of checkpoint folder>")`
as an option to use a locally stored checkpoint of their model for inference.

### Expected behavior

Following guide as written should not result in crash.

Additionally, users should know how to use the model when it is saved locally (as checkpointing automatically does). 

Comments:
Very related to  #30058
Support mixed-language batches in `WhisperGenerationMixin`
Author: cifkao
Pull request: no
Labels: name: Feature request

name: Audio

### Feature request

It is currently not possible to mix multiple languages in a single batch when running [Whisper](https://huggingface.co/docs/transformers/en/model_doc/whisper). The `language` argument only accepts a single string (as opposed to a separate language for each batch item), and if no language is passed and multiple languages are detected, [transcription will fail](https://github.com/huggingface/transformers/blob/5011908e10d9592eeb634f4940e0bc130d3edc69/src/transformers/models/whisper/generation_whisper.py#L1170-L1173).

I propose to enable passing a list of languages (`language: Optional[Union[str, List[str]]]`) in a batched transcription situation, as well as removing the restriction related to language detection.

### Motivation

Not being able to transcribe multiple languages in a single batch is clearly a limitation, especially when relying on auto-detection, but also in scenarios where the language is known.

The [error message](https://github.com/huggingface/transformers/blob/5011908e10d9592eeb634f4940e0bc130d3edc69/src/transformers/models/whisper/generation_whisper.py#L1172) states that `It is currently not supported to transcribe to different languages in a single batch.`, implying that it could be supported at some point.

### Your contribution

I have implemented this and I'm planning to submit a PR.

Comments:
cc @ylacombe @sanchit-gandhi
why cuda.amp was removed from version 3.35.0
Author: thbupt
Pull request: no

### System Info

transformers==3.35.0 and transformers==3.34.0

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

In transformers==3.35.0:
![image](https://github.com/huggingface/transformers/assets/4949487/8258fd71-ac90-4bde-b693-8a57e83b4806)
In transformers==3.34.0
![image](https://github.com/huggingface/transformers/assets/4949487/76ea8383-c61d-49c5-a03f-72847f823923)

### Expected behavior

It seems that you can only use  Mixed precision training with cpu_amp or apex, while torch.cuda.amp was removed.

Comments:
Hi @thbupt, you can find the relevant PR here: https://github.com/huggingface/transformers/pull/25702

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
ImportError: cannot import name 'get_model_param_count' from 'transformers.trainer_pt_utils'
Author: keyuchen21
Pull request: no

### System Info

- `transformers` version: 4.23.1
- Platform: Linux-5.15.146.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.9.19
- Huggingface_hub version: 0.22.2
- PyTorch version (GPU?): 2.2.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

import transformers
print(transformers.__version__)
from transformers.trainer_pt_utils import get_model_param_count

python version: 3.9

### Expected behavior

<img width="768" alt="image" src="https://github.com/huggingface/transformers/assets/54015474/c719a237-0499-4e88-97c0-15fae9c1568a">


Comments:
I guess `get_model_param_count` was not available in Transformers v4.23.1. Hence you can upgrade with `pip install --upgrade transformers`.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Remove unused module DETR based models
Author: conditionedstimulus
Pull request: yes

# What does this PR do?
Removing the unused classification head module from object detection models primarily based on DETR.

Fixes # (issue)
- Conditional DETR (copied from DETR)
- Deformable DETR  (copied from DETR)
- DETA  (copied from DETR)
- DETR
- Table Transformer  (copied from DETR)


## Who can review?
@amyeroberts @NielsRogge 


Comments:
Video-LLaVA with transformers library
Author: Kamakshi8104
Pull request: no
Labels: name: New model

### Model description

Video-LLaVA is a multimodal model that is trained on both images and videos simultaneously.  I feel that it would be highly beneficial if it is added to the transformers library as it is a pretty good choice when it comes to video-question answering

### Open source status

- [ ] The model implementation is available
- [ ] The model weights are available

### Provide useful links for the implementation

https://huggingface.co/LanguageBind/Video-LLaVA-7B 


Comments:
If the authors will not be willing to contribute, I want to work on this

cc @gante 

Great @zucchini-nlp! I'd reach out on the model page on the hub, asking the authors if they'd like to add it to the library. If they're happy to just leave as-is it's all yours!
Jamba - Skip 4d custom attention mask test
Author: amyeroberts
Pull request: yes

# What does this PR do?

Fixes failing test on main: https://app.circleci.com/pipelines/github/huggingface/transformers/92832/workflows/0cbde306-131c-479b-b938-bca2fb838a82/jobs/1215777

Applies the same fix as #30780

However, applying created another failing test: `test_assisted_decoding_matches_greedy_search_0_random`. 

This fails because the model now has `_supports_cache_class` unset, but passes `past_key_values` as `HybridMambaAttentionDynamicCache`, which [triggers an exception here](https://github.com/huggingface/transformers/blob/a42844955f3134a2b2fd8075353c710dcfb7b362/src/transformers/generation/utils.py#L1105). 

Mamba doesn't have this issue, because it doesn't define any models in `all_generative_models` in its model tester, and so the generation tests "pass" but are in fact skipped.  

The tests are skipped now, but we should update either the interaction between the cache and generation to ensure this isn't necessary. I've opened a related issue here #30828 

cc @gante 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30826). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Pin deepspeed properly
Author: muellerzr
Pull request: yes

# What does this PR do?

Pins deepspeed in the dep table to (hopefully) fix the test failures

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ydshieh @pacman100 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30801). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I don't feel strong between these 2 choices. Disable them means we can't see any failure (if any) introduced by our code changes.
And I don't know if it's a (little) bit time consuming to to skip those 10 tests after all the work done by now.
But I will leave you two to make the final call.

Closing in favor of Sourab's fix :)
Deprecate models script - correctly set the model name for the doc file
Author: amyeroberts
Pull request: yes

# What does this PR do?

There was a bug in #30184, in initial model name e.g. `model_name` was returned as the found model doc name, instead of the one which relates to the found path. 

This means, if the model name has a different hyphenation, it's config cannot be found, and therefore it is not deprecated. 

This fixes the issue for certain models e.g. `gptsan_japanese`. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30785). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ydshieh There was actually one remaining bug! I didn't catch because when I tested I wasn't using models with hyphens in the name. 

> I am not sure if we have a correspondence of model_name to model_doc_path.

@ydshieh Ah, yes, it's a bit unclear from this diff. The model name returned here is set to [model_doc_name here](https://github.com/huggingface/transformers/blob/bdfefbadaf7553d8a104dce096fd2905692f8197/utils/deprecate_models.py#L99). So the correspondence should work, because the mapping is just between the name used and the doc path. 

@amyeroberts 

right, thanks!
Update ds_config_zero3.json
Author: pacman100
Pull request: yes

# What does this PR do?
1. Instead of pinning deepspeed and making a lot of changes to the self-hosted runners in https://github.com/huggingface/transformers/pull/30801, this disables CPU offloading which are leading to 12 test failures. This PR needs to be reverted once the DeepSpeed fixes the issue: https://github.com/microsoft/DeepSpeed/issues/5538

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30829). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @amyeroberts 

Thank you for all of you!
Load nf4 weights/model in bfloat16
Author: RonanKMcGovern
Pull request: no
Labels: name: Quantization

### Feature request

Allow quantized models to be loaded (upcasted) to bfloat16.

### Motivation

1. The big benefit of QLoRA is being able to fine-tune on less VRAM AND with much less time than full fine-tuning.
2. Adding in rsLoRA, initiating LoRA adapters to minimise quanting error,  and training norm and embedding layers gets performance close to full fine-tuning.

HOWEVER, there's a catch that it's not possible to get back to a bf16 model (yes, it's possible to merge to 4-bit, but this hurts accuracy, and there's no way to upcast either). See [this issue](https://github.com/huggingface/peft/pull/851).


### Your contribution

BEST approach:
- Allow a type of merge where a quantized base model is merged with LoRA weights to form a 16 bit (float16 or bfloat16) model. [Obviously this will take more RAM or VRAM, so may not suit certain users, but it is best for quality].

HACKY but sufficient approach:
- Simply allow a quantized model (i.e. saved like [this](https://github.com/huggingface/transformers/pull/26037)] to be loaded in bfloat 16. If that's possible, then the following flow would work:
1. Fine tune a model with QLoRA (ideally making norms and embeds trainable). [initialise the adapters to minimise quanting error]
2. Save adapters.
3. Unload the adapters and save the nf4 base model (ideally including the trained norms and embeds)
4. Re-load the nf4 base model, but upcasted to bfloat16.
5. Apply the adapter
6. Merge the adapter to the base model
7. This should result in a highly accurate final model.

Everything 1-7 is possible already except for step 4. However, step 4 is done in unsloth, see [here](https://github.com/unslothai/unsloth/issues/195), so should be fairly easy.

[Side note - I'm unsure whether bnb quants the norm and embed layers, or just the linear layers?]

Comments:
@younesbelkada @SunMarc 

Thanks @RonanKMcGovern for this feature request!
I think this makes sense ! I just have one question:
for the step 4-, why users couldn't simply load again the base model but directly in bf16 ? What would be the benefit of un-quantizing on the fly a 4bit model vs loading it directly in 16bit ? Is it to preserve the quantization error?

> for the step 4-, why users couldn't simply load again the base model but directly in bf16 ? What would be the benefit of un-quantizing on the fly a 4bit model vs loading it directly in 16bit ? Is it to preserve the quantization error?

Yeah, a base bf16 model will have slightly different weights than a quantized and then dequantized bf16 model. This makes a performance difference when applying and merging the 16 bit adapters on top.

Make sense?

Makes sense ! 
@RonanKMcGovern I made: https://github.com/huggingface/transformers/pull/30806 would that PR work for you? tested it locally and it seems to produce coherent text when doing text generation. API-wise it should be quite straightforward (see the PR description)