noamaanMulla-03/transformers-issues huggingface dataset:
VideoLLaVa: fix chat format in docs
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

As per title, this PR fixes some typos. I added chat template in the hub repo, but my PR wasn't merged yet. So, let's not add here examples with chat template yet

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32083). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Chameleon, Hiera] Improve docs
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR improves the docs of:
* Chameleon: make sure a working checkpoint is displayed, move to multimodal section
* Hiera: adds a figure, adds resources

cc @zucchini-nlp @EduardoPach 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32038). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix tests skip
Author: qubvel
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Some tests have `self.skip` instead of `self.skipTest`

## Who can review?

@amyeroberts 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32012). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: Fixed incorrect dictionary assignment in `src/transformers/__init__.py`
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
Just a small fix, replaced the colon `:` with an equal `=` sign for proper dictionary assignment.

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts

Comments:
Bug report update
Author: gante
Pull request: yes

# What does this PR do?

We often get issues that are usage questions. Ignoring issues is not cool, taking the time to reply with information that is in the docs is even less cool (we could be spending that time for everyone's benefit instead).

This PR proposes the following changes:
1. Our contribution guide, in the bug reporting section, points to our new [docs bot](https://huggingface.co/spaces/huggingchat/hf-docs-chat)
2. Our new issue page points to the bug reporting section of the contribution guide **right at the top**

Hopefully this could lower the number of "non-issues" we get in `transformers`, while empowering the users to be more self-sufficient 💪 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31983). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Generate: remove deprecated code due to `Cache` and `cache_position` being default
Author: gante
Pull request: yes

# What does this PR do?

1. Simplifies `prepare_inputs_for_generation` on models using `Cache`
2. Removes the unused `_reorder_cache` function on models using `Cache`

____________________________________________
Slow tests run (and passing / same failures as in `main`):
- [x] (`Cache` integration tests) `RUN_SLOW=1 py.test -vv tests/utils/test_cache_utils.py`
- [x] (`generate` integration tests) `RUN_SLOW=1 py.test -vv tests/generation/test_utils.py`
- [x] (reference LLM model, `llama`) `RUN_SLOW=1 py.test -vv tests/models/llama/test_modeling_llama.py`
- [x] (reference MoE model, `mixtral`) `RUN_SLOW=1 py.test -vv tests/models/mixtral/test_modeling_mixtral.py`
- [x] Slow tests for ALL other models in the diff


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31898). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker ready for a final check :) (tons of slow tests ran on my end, should be safe to merge)

> LGTM then! i trust our tests for this, would be nice to see the results of the full suit! cc @ydshieh

Do you want me to trigger a full (GitHub Action) CI for this PR during this weekend (before merge)?

yeah would be nice ! 

OK  will do (once today's CI run is over)

@ydshieh please ping me when the run is over 🤗 

btw, there are MANY broken tests in the list of models changed in this PR (on `main`), mostly SDPA, FA2, and integration tests :o I should work on it 🤔 

@gante 

The run (triggered for this PR) is likely to be over tomorrow morning (if I trigger it this evening). I will let you know in any case.

CI is running [here](https://github.com/huggingface/transformers/actions/runs/9910094178)



slow CI looks good (same issues as in `main`), merging 🤗
stop_strings not work at all
Author: lucasjinreal
Pull request: no
Labels: name: Generation

    reversed_stop_string = stop_string[::-1]
TypeError: 'NoneType' object is not subscriptable


output_ids = model.generate(
                input_ids,
                images=images_tensor,
                do_sample=False,
                temperature=0.2,
                max_new_tokens=1024,
                streamer=streamer,
                use_cache=True,
                # cache_position=None,
                pad_token_id=tokenizer.eos_token_id,
                # pad_token_id=conv.stop_token_ids,
                eos_token_id=conv.stop_token_ids,
                # stopping_criteria=[stopping_criteria],
                tokenizer=tokenizer,
                stop_strings=[conv.stop_str]
            )

Comments:
Hey! It will be helpful if you provide an executable code snippet to reproduce the error :)

cc @Rocketknight1 

Yes, this seems like `conv.stop_str` is `None`? Are you sure you're passing that correctly?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
T5 Static Cache
Author: huseinzol05
Pull request: yes

This enable to use torch.compile for T5 generation to enable faster generation.

Compiled static cache able to achieve 905.746397513013 tokens / sec while non-compiled got 259.57796367162115 tokens / sec on Small T5 60M parameters.

1. Notebook for non-compiled, https://github.com/mesolitica/t5-static-cache/blob/main/t5-static-cache-non-compile.ipynb
2. Notebook for compiled, https://github.com/mesolitica/t5-static-cache/blob/main/t5-static-cache.ipynb

## Still work in progress

1. Current forked only work to use static cache, need to follow caching steps as Llama.
3. There are so many conditions need to fulfill first.
4. Only worked on Pytorch 2.4.0.dev20240508+cu121 version, not yet released as stable for custom function reduce-overhead torch compile.

Comments:
cc @ArthurZucker 

Once this is ready feel free to ping either @fxmarty or @gante 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
`num_input_tokens_seen` included the `pad` tokens if sample padding strategy used 
Author: thincal
Pull request: no

### System Info

latest transformers

### Who can help?

@muellerzr @pacman100

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

If training sample is batched with padding strategy, the num_input_tokens_seen will be calculated including the `pad` tokens, this is not expected.

### Expected behavior

1. Excluding the `pad` tokens during the calculation of `num_input_tokens_seen`, a suggested fix as bellow:
```
inputs_device = "cuda" if self.args.distributed_state.backend == "nccl" else "cpu"

self.state.num_input_tokens_seen += torch.sum(
    self.accelerator.gather(
        torch.tensor(torch.sum(inputs[main_input_name] != self.tokenizer.pad_token_id).item(), device=inputs_device, dtype=torch.int64)
    )
).item()
```

2. And if the nccl backend used but `input_ids` stay in cpu it will also [cause problem](https://github.com/huggingface/transformers/issues/28791#issuecomment-1921816863), this is also fixed by deciding the device type according to backend.
```
inputs_device = "cuda" if self.args.distributed_state.backend == "nccl" else "cpu"
```

Comments:
@pacman100 could help review with this issue reported ? thanks!

Gentle ping @muellerzr @pacman100

cc @muellerzr @SunMarc 

Hi @thincal, thanks for raising the issue and sorry for the delay ! This seems like a good idea to remove the padded tokens and change the device depending on the backend ! Would you like to open a PR ? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
past_key_values for SeamlessM4Tv2ForSpeechToText is not working as expected
Author: sashulya667
Pull request: no
Labels: name: bug

name: Audio

### System Info

transformers version: 4.37.2
python verison: 3.8.6.
OS: Windows 11

### Who can help?

@sanchit-gandhi

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I have segments of audio, and I would like to pass past_key_values between them. I exptected the transcribtion quality to increase, but rather it became unreadable.

```python

processor = AutoProcessor.from_pretrained(path_to_model)
model = SeamlessM4Tv2ForSpeechToText.from_pretrained(path_to_model)
audio_chunks = [audio_segments]

past_key_values= None

for i in range(5):
    audio_inputs = processor(audios=audio_chunks[i], return_tensors="pt", sampling_rate=16_000)

    output = model.generate(**audio_inputs, tgt_lang="rus", repetition_penalty=1.1, return_dict_in_generate=True, past_key_values=past_key_values)
    
    tmp_result = processor.decode(output[0][0], skip_special_tokens=True)
    past_key_values = output['past_key_values']

```

### Expected behavior

The transcription quality is supposed to increase when I pass past_key_values (or at least have similar transcribation quality as when past_key_values=None).
The audio is the same. In other words, I had some audio, applied VAD to segment it into batches, and then fed this segments to the model one by one.

Comments:
cc @ylacombe 

Gentle ping @sanchit-gandhi @ylacombe 

Another ping @sanchit-gandhi @ylacombe 

Hey @vapemaster-kz, I believe you're misunderstanding what past-key values are used for!

You can find on the docs [here](https://huggingface.co/docs/transformers/model_doc/seamless_m4t_v2#transformers.SeamlessM4Tv2ForSpeechToText.forward.past_key_values) that past-key values `can be used to speed up sequential decoding`.

Past key values are simply a way to avoid re-computing the same information over and over again when calling `generate` from scratch *on the same generation pass*.

I hope it helps!


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey @vapemaster-kz, did my last comment solved your issue? Let me know! 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add image-text-to-text task guide
Author: merveenoyan
Pull request: yes

Added shortly `image-text-to-text` task guide that includes streaming and more 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31777). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts I have addressed all your comments, can you merge if you approve?

@merveenoyan Not really sure why the CI runs are consistently failing here, but since this PR is just a doc page so shouldn't affect hub etc. I'm going to merge
ValueError: The checkpoint you are trying to load has model type `chameleon` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.
Author: KaifAhmad1
Pull request: no
Labels: name: bug

### System Info

Windows 11x64 
Cuda: 12.1 
Python 3.10.10 
transformers 4.42.4


### Who can help?

Hey, @amyeroberts, could you help with this issue?

Thanks!
Kaif

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```  Python 

!pip install -qU transformers einops datasets bitsandbytes

from transformers import AutoModelForCausalLM, AutoProcessor
import torch

device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

model = AutoModelForCausalLM.from_pretrained("facebook/chameleon-7b", trust_remote_code=True).to(device)
processor = AutoProcessor.from_pretrained("facebook/chameleon-7b", trust_remote_code=True).to(device)

```



``` 
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    981             try:
--> 982                 config_class = CONFIG_MAPPING[config_dict["model_type"]]
    983             except KeyError:

3 frames
KeyError: 'chameleon'

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/transformers/models/auto/configuration_auto.py in from_pretrained(cls, pretrained_model_name_or_path, **kwargs)
    982                 config_class = CONFIG_MAPPING[config_dict["model_type"]]
    983             except KeyError:
--> 984                 raise ValueError(
    985                     f"The checkpoint you are trying to load has model type `{config_dict['model_type']}` "
    986                     "but Transformers does not recognize this architecture. This could be because of an "

ValueError: The checkpoint you are trying to load has model type `chameleon` but Transformers does not recognize this architecture. This could be because of an issue with the checkpoint, or because your version of Transformers is out of date.

```

### Expected behavior

It will load the model without raising any error 

Comments:
Hi,

See https://github.com/huggingface/transformers/issues/32098. Also make sure to install Transformers from source for now: `pip install git+https://github.com/huggingface/transformers.git`.

It will be included in the next PyPi release.
[RT-DETR] Add resources
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds resources for RT-DETR as well as a figure.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31815). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add conversion for interleave llava
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Adds conversion script for the models proposed in [this blog post](https://llava-vl.github.io/blog/2024-06-16-llava-next-interleave/). 

**Briefly**: Llava-Next_Interleave is a series of models trained with interleaved visual inputs, including video and 3D images. Even though it's a lllava-next series, the authors have hardcoded the inference script to not use anyres technique, so I added it in LLaVa. We will not support video natively as LLaVaNeXTVideo, but user can pass each frame separately as if it's an interleaved image input 

The main idea was to support the 0.5B checkpoint, because it has same performance as llava-next-7b in the blog post. Imo it can be a cool addition  

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31858). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
SpeechEncoderDecoder doesn't support param buffer assignments
Author: muellerzr
Pull request: yes

# What does this PR do?

An encoder/decoder model I somehow missed, needs the `_supports_param_buffer_assignments=False` flag :)

(Solves red on main)

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32009). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix if else and *actually* enable superfast init
Author: muellerzr
Pull request: yes

# What does this PR do?

Borked part of the refactor from the PR, the logic should default to `True` not `False`, otherwise it's not enabled for any model! 😱 

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32007). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix: Fixed the arguments in `create_repo()` function call
Author: Sai-Suraj-27
Pull request: yes

# What does this PR do?
The argument names passed in the `create_repo()` function call here are currently wrong, we can see this from the actual definition of [create_repo() here](https://github.com/huggingface/huggingface_hub/blob/982f9a2b649dbef183857fa643d3c80ddd3ee8b3/src/huggingface_hub/hf_api.py#L3239), there are no arguments `name` & `organization`.
https://github.com/huggingface/transformers/blob/7f79a97399bb52aad8460e1da2f36577d5dccfed/src/transformers/commands/user.py#L188

Found it similar to [this class](https://github.com/huggingface/huggingface_hub/blob/982f9a2b649dbef183857fa643d3c80ddd3ee8b3/src/huggingface_hub/commands/user.py#L127).

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts @ArthurZucker 

Comments: