noamaanMulla-03/transformers-issues huggingface dataset:
Add video text to text docs
Author: merveenoyan
Pull request: yes

Adding video-text-to-text task guide

@zucchini-nlp @NielsRogge @amyeroberts @stevhliu 

Comments:
There seems to be an issue with Llava, in the latest stable some image tokens are cropped, i.e. when I pass 12 downsampled frames and 12 <image> tokens it tells me to add 13 image tokens instead. It's fixed in main, but now I get `
TypeError: LlavaForConditionalGeneration.forward() got an unexpected keyword argument 'num_logits_to_keep'` from `result = self._sample(input_ids, logits_processor=prepared_logits_processor)`.

Caused by https://github.com/huggingface/transformers/pull/31292, will work on it

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33164). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@zucchini-nlp re: slack discussions I'd say we merge this and edit when the processors are out.
Incorrect logits shape for GIT model (microsoft/git-base-textvqa)
Author: souradipp76
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.42.4
- Platform: Linux-6.1.85+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.23.5
- Safetensors version: 0.4.4
- Accelerate version: 0.32.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.3.1+cu121 (True)
- Tensorflow version (GPU?): 2.17.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.8.4 (gpu)
- Jax version: 0.4.26
- JaxLib version: 0.4.26
- Using distributed or parallel set-up in script?: no
- Using GPU in script?: no
- GPU type: Tesla T4

### Who can help?

@ArthurZucker @amyeroberts @zucchini-nlp

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am trying to use GIT multimodal model (microsoft/git-base-textvqa) for Visual Question Answering. The shape of logits from the output of the forward function is not (batch_size, sequence_length, config.vocab_size) as mentioned in the documentation ([GIT](https://huggingface.co/docs/transformers/main/en/model_doc/git#transformers.GitForCausalLM.forward)).

1. Run the following code snippet
```
from transformers import AutoProcessor, AutoModelForCausalLM
from huggingface_hub import hf_hub_download
from PIL import Image

processor = AutoProcessor.from_pretrained("microsoft/git-base-textvqa")
model = AutoModelForCausalLM.from_pretrained("microsoft/git-base-textvqa")

file_path = hf_hub_download(repo_id="nielsr/textvqa-sample", filename="bus.png", repo_type="dataset")
image = Image.open(file_path).convert("RGB")

pixel_values = processor(images=image, return_tensors="pt").pixel_values

question = "what does the front of the bus say at the top?"

input_ids = processor(text=question, add_special_tokens=False).input_ids
input_ids = [processor.tokenizer.cls_token_id] + input_ids
input_ids = torch.tensor(input_ids).unsqueeze(0)
print(input_ids.shape) # torch.Size([1, 13])

output = model(pixel_values=pixel_values, input_ids=input_ids)
logits = output.logits
print(logits.shape) # torch.Size([1, 914, 30522])
```

### Expected behavior

The shape of the output logits should be (batch_size, sequence_length, config.vocab_size) where shape of input_ids is  (batch_size, sequence_length).

print(input_ids.shape) # torch.Size([1, 13])
...
print(logits.shape) # torch.Size([1, 13, 30522])

Comments:
Hi,

The sequence_length which comes out of GIT are the image tokens and text tokens concatenated, along with 1 special CLS token. So in this case, the 914 comes from 13 text tokens, and 914 - 13 - 1 = 900 image tokens (which you can also see by doing (`model.config.vision_config.image_size` // `model.config.vision_config.patch_size`)**2)

Thanks @NielsRogge for the clarification.
Add duckduckgo search tool
Author: aymeric-roucher
Pull request: yes
Labels: name: Agents

# What does this PR do?

Add a search tool based on Duckduckgo API (does not need an API key).

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32882). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix: Suppressed 'use_reentrant=False' warning
Author: ankush13r
Pull request: yes

# What does this PR do?

Fixes #28536 Resolving 'use_reentrant=False' warning due to empty dict in gradient_checkpointing_kwargs.

Referenced prior issue where `gradient_checkpointing_kwargs` was initialized as an empty dict in the Trainer. This led to the `gradient_checkpointing_enable` function not handling it as `None`, causing a 'use_reentrant=False' warning. The issue was resolved by removing the unnecessary initialization, ensuring proper handling of gradient checkpointing parameters. 

For more information, see: https://github.com/huggingface/transformers/issues/28536#issuecomment-2312997910

@ArthurZucker

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33208). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Failed to import transformers.pipelines: No module named 'distutils'
Author: Rajan-Thakur
Pull request: no

## Issue:
```
Exception has occurred: RuntimeError
Failed to import transformers.pipelines because of the following error (look up to see its traceback):
No module named 'distutils'
ModuleNotFoundError: No module named 'distutils'
```
After importing and attempting to use the "transformers.pipeline" function, the above error occurs.

## System information

- Python version: 3.12.4
- Pip version: 24.2 (up-to-date)
- Transformers version: 4.43.4
- PATH variables all set correctly

## Notes:

- Importing "transformers" in itself wasn't an issue, but when attempting to use specifically the "transformers.pipeline" function, it fails and shows the error.
- I have tried installing "distutils" through "pip" and through "pip3," but it has repeatedly displayed a message saying no such package exists at all.
- IMPORTANT: I know that "distutils" is a deprecated package. I've tried installing "setuptools," but that hasn't solved the issue. The same exact error message shows.

So, what now?

Comments:
Hi @Rajan-Thakur, thanks for reporting an issue! 

> PATH variables all set correctly
This is quite a general statement, which we can't verify here with the current information. It does seem there's something funny happening in the env. 

Could you try the following in a python environment: 
```py
import setuptools
print(setuptools.__version__)
```

Version is 72.1.0

So what now?

Actually never mind, thank you @amyeroberts anyway, I solved it myself somehow, by just
`python3 -m pip uninstall setuptools`
`python3 -m pip install setuptools`
executing that in the command prompt.

For future dwellers who see and try this, but it doesn't work for you, try re-installing transformers and torch too.

Hey, any reason this was re-opened? 

Just in case if anyone has a similar issue and has a question. I'll close it now.
docs: Replace package abbreviations with full name(`bitsandbytes`) in docstrings
Author: rapsealk
Pull request: yes

# What does this PR do?
Hello, transformers team!

While setting up a new environment for LLM fine-tuning, I encountered an error message stating: `Trainer tried to instantiate bnb optimizer but bnb is not installed!`. I recognized that `bnb` refers to the `bitsandbytes` package and resolved the issue by installing `bitsandbytes`. However, this could be confusing for others who might not be aware that `bnb` stands for `bitsandbytes`.

To improve clarity and help users identify and install the correct package, I propose updating the docstrings to replace abbreviations like `bnb` with the full name `bitsandbytes`. This change will provide clearer guidance and reduce potential confusion when encountering similar error messages or documentation references.

Thanks for the review in advance!

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Fix synced GPUs
Author: gante
Pull request: yes

# What does this PR do?

do not merge -- this is a sample of a working solution, to be properly addressed as part of #32685 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33252). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Chat Assistant Prefill
Author: pseudotensor
Pull request: no
Labels: name: Feature request

### Feature request

https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering/prefill-claudes-response
https://www.anthropic.com/news/claude-2-1-prompting

I expected I could prefill the assistant response, but seems like it doesn't work.

I should be able to do:
```
messages = [
    {
        "role": "user",
        "content": prompt,
    },
        "role": "assistant",
        "content": 'According to ',
    }
]
```
And it should use this to generate up through assistant's response but not ending it, so the model continues

Anthropic has this feature, and it helps to control the responses.

Yes, one can avoid the chat API, but since the chat template work is so pervasive and useful, it would be great to add this extension.

### Motivation

Control model output more precisely.  See web sites above.

### Your contribution

I'm unclear if it's even possible within the general chat framework.  Maybe the jinja2 template would support it OOTB or not, I'm not sure how much it depends upon the template writer, but even if one had to tweak an existing template, still would require the chat API to handle.

Comments:
cc @Rocketknight1 

Hi @pseudotensor! You're right that chat templating as written right now doesn't support this, although I agree it would definitely be useful. 

I can see three ways to do this:

1) We use the existing `add_generation_prompt` flag, but we change the templates so that if the user sets `add_generation_prompt` **and** the last message is an `assistant` message, then instead of a normal generation prompt, the template leaves the final assistant message "unfinished" without an `<eos>` or `<eot>` token.

2) We add a new flag, like `continue_final_message` or something like that, which also has the same effect of not adding any end tokens to the final assistant message.

3) We add a hacky option to just trim special tokens from the end of the output rendered by the template.

Options 1 and 2 are obviously more principled, but we'd have to update existing templates to support them. 3) is much hackier, but it has the advantage that we wouldn't need to update any templates.

WDYT - do you have any strong preferences here? I think I lean towards 1 personally, but I'm a little worried about the risk of unexpected side effects.

@pseudotensor with apologies for the delay, a PR to add this is now up at #33198!
Enable multi-device for Segformer
Author: Nech-C
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #29786  (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@amyeroberts

Comments:
As [discussed here](https://github.com/huggingface/transformers/issues/29786#issuecomment-2313050997), waiting for wider discussion about weight allocation from accelerate, as segformer will never be correctly split across devices due to highly imbalance sizes of different model modules
Add warm up in evaluation_loop to get more accurate performance data
Author: zxd1997066
Pull request: yes

Add warm up in evaluation_loop, this will make performance metrics more accurate when using torch_compile, take `text-classification+albert-base-v1` as example:
`python -u ./transformers/examples/pytorch/text-classification/run_glue.py --model_name_or_path albert-base-v1 --task_name MRPC --do_eval --max_seq_length 16 --learning_rate 2e-5 --overwrite_output_dir --output_dir /tmp/tmp_huggingface/ --torch_compile --report_to=none`
before this PR:
```
***** Running Evaluation *****
[INFO|trainer.py:3831] 2024-08-12 07:22:17,390 >>   Num examples = 408
[INFO|trainer.py:3834] 2024-08-12 07:22:17,390 >>   Batch size = 8
100%|████████████████████████████████████████████████████████████████████████| 51/51 [00:00<00:00, 69.95it/s]
***** eval metrics *****
  eval_accuracy               =     0.5637
  eval_combined_score         =     0.6024
  eval_f1                     =     0.6411
  eval_loss                   =     0.6855
  eval_model_preparation_time =     0.6604
  eval_runtime                = 0:00:06.56
  eval_samples                =        408
  eval_samples_per_second     =     62.169
  eval_steps_per_second       =      7.771
```
after this PR:
```
***** Running Evaluation *****
[INFO|trainer.py:3831] 2024-08-12 07:11:18,818 >>   Num examples = 408
[INFO|trainer.py:3834] 2024-08-12 07:11:18,818 >>   Batch size = 8
100%|██████████| 51/51 [00:00<00:00, 68.18it/s]
***** eval metrics *****
  eval_accuracy               =     0.5637
  eval_combined_score         =     0.6024
  eval_f1                     =     0.6411
  eval_loss                   =     0.6855
  eval_model_preparation_time =     6.4489
  eval_runtime                = 0:00:00.76
  eval_samples                =        408
  eval_samples_per_second     =    533.526
  eval_steps_per_second       =     66.691
```

Comments:
Hi @zxd1997066, thanks for opening a PR! 

How "accurate" the measurement is here depends on all sorts of situations -- for people not compiling the model this add the overhead of a forward pass, which isn't representative of preparation time -- and so adding would both be misleading for some users and would be a breaking change as users reported metrics to date would suddenly change. 

If this is something you want to include - subclassing trainer and overriding the evaluation loop is a quick way to incorporate into your own evals!
Generate: fix assistant in different device
Author: gante
Pull request: yes

# What does this PR do?

See title :)

Fixes `test_assisted_decoding_in_gpu_cpu` (renamed in this PR to a more descriptive name)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33257). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
DETR documentation typo: repeated word `be be`
Author: sergiopaniego
Pull request: no

In [DETR documentation - Usage tips](https://huggingface.co/docs/transformers/v4.44.2/en/model_doc/detr#usage-tips), there is a repeated word in the sentence:

````
These can be be provided to either CocoEvaluator or PanopticEvaluator, which allow you to calculate metrics like mean Average Precision (mAP) and Panoptic Quality (PQ).
````


The word `be` is repeated.



Comments:
Add GraniteRMSNorm
Author: NielsRogge
Pull request: yes
Labels: name: run-slow

# What does this PR do?

This PR is a follow-up of #31502 which broke Transformers for PyTorch < 2.4.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33177). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi this makes sense.
I had added the `nn.RMSNorm` class
but looks like there is some issue with fp16 as well: https://github.com/pytorch/pytorch/pull/134106


Should I run slow tests or can this be merged as-is?

might be better to run slow tests on the granite class @NielsRogge 

Seems like the slow tests are failing (cc @ydshieh), but I assume it's safe to merge this PR since the following passes

```python
from torch import nn
import torch


class GraniteRMSNorm(nn.Module):
    def __init__(self, hidden_size, eps=1e-6):
        """
        GraniteRMSNorm is equivalent to T5LayerNorm
        """
        super().__init__()
        self.weight = nn.Parameter(torch.ones(hidden_size))
        self.variance_epsilon = eps

    def forward(self, hidden_states):
        input_dtype = hidden_states.dtype
        hidden_states = hidden_states.to(torch.float32)
        variance = hidden_states.pow(2).mean(-1, keepdim=True)
        hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
        return self.weight * hidden_states.to(input_dtype)

    def extra_repr(self):
        return f"{tuple(self.weight.shape)}, eps={self.variance_epsilon}"
    

a = torch.nn.RMSNorm(10)

b = GraniteRMSNorm(10)

assert a.weight.shape == b.weight.shape

c = torch.randn(1, 10)

assert torch.allclose(a(c), b(c))
```

when will this release to fix transformers?
creating lot of conflicts as pytorch==2.4 is not supported by many packages like vllm, flash_attn, etc.

@ArthurZucker can you merge this?
I will create a new PR for fixing the slow tests.
Looks like its blocking the DeepSpeed team.

Okay

cc @ydshieh if you see failures on this, it's expected! 

Thanks @ArthurZucker , Ill fix this test in a new PR
Fix: use `torch.from_numpy()` to create tensors for np.ndarrays
Author: shinyano
Pull request: yes

# What does this PR do?

Fixes #33185 by using `torch.from_numpy()` to create tensors for np.ndarrays, instead of `torch.tensor()` which may cause hanging.

I did some rough search in the code and made changes where I thought necessary. Code using `tensor()` with `dtype` specified remains unchanged to prevent potential type errors.

Also, `torch.from_numpy()` would share memory with numpy array, while `torch.tensor()` copies data. If this change would cause potential problems, please let me know.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@qubvel Thanks for your review!



Comments:
remove torch input dependant control flow
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fix granite slow compile tests. 
https://github.com/huggingface/transformers/actions/runs/10656977518/job/29536387749


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33245). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix: `num_logits_to_keep` in composite models
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes issue from https://github.com/huggingface/transformers/pull/33164#issuecomment-2315111463. 

We cannot assume that the model will have `num_logits_to_keep` in its forward, because an LM can be used as backbone in composite models. So the current PR sets `num_logits_to_keep` to None by default and adds it as a kwarg only when it is explicitly set to an integer by users

Plus, updated VLMs accordingly with `num_logits_to_keep`. Other VLMs like BLIP don't need this arg because they delegate the whole generation process to the LM.

cc @merveenoyan 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33168). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Would it be possible to add a test to ensure no regression on the issue https://github.com/huggingface/transformers/pull/33164#issuecomment-2315111463 ?

Oh, also cc @gante for generation :)

@LysandreJik yes, I am working on enabling generation tests on all VLMs. That should be 2-3 PRs away, after we fix some issues after the latest refactoring on VLM processing code

We've overlooked many issues on VLM side due to the absence of fast generation tests and slow tests are usually not ran if modeling code is unchanged.