noamaanMulla-03/transformers-issues huggingface dataset:
Generate: remove deprecated public decoding functions and streamline logic 🧼 
Author: gante
Pull request: yes

# What does this PR do?

🧼 🧼 🧼 

Calling the internal decoding functions as part of our public API was scheduled for removal in v4.41 (the next release). Its motivation was flexibility and conciseness: having multiple public interfaces for the same functionality forced us to add repeated logic in many places, increasing every time we added a new decoding method.

Due to this removal from the public API, a few things were changed/removed as a logical consequence:
1. No more documentation/examples in the internal decoding functions. We have a [page](https://huggingface.co/docs/transformers/generation_strategies#decoding-strategies) with examples to call the internal decoding methods from `generate`;
2. The arguments to the decoding functions are no longer optional -- the decoding functions are exclusively called from `generate`. As such, we can remove a lot of boilerplate (`x = x if x is not None else self.generation_config.x`).
3. No more tests regarding the internal decoding methods -- already removed in [this PR](https://github.com/huggingface/transformers/pull/29582)

______________________________________________________________________
Tests ran locally:
- [x] generate doctests (`pytest --doctest-modules src/transformers/generation -vv`)
- [x] generate integration tests (`RUN_SLOW=1 py.test tests/generation/test_utils.py -vv`)
- [x] cache integration tests (`RUN_SLOW=1 py.test tests/test_cache_utils.py -vv`) -- same failures as in `main`
- [x] llama slow tests (`RUN_SLOW=1 py.test tests/models/llama/test_modeling_llama.py -vv`)
- [x] whisper slow tests (`RUN_SLOW=1 py.test tests/models/whisper/test_modeling_whisper.py -vv`) -- same failures as in `main`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29956). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Tests on all models are passing 🙌 (the failing pipeline test seems unrelated, and passing locally on my end)

ping @ArthurZucker -- ready for review :)

Reran slow tests locally, all seems good 👍
TimeSeriesTransformerForPrediction model unused parameters Runtime error in Distributed environment
Author: maeschbacher
Pull request: no
Labels: name: Good Difficult Issue

### System Info

transformers==4.31
accelerate==0.21

I'm trying to run TimeSeriesTransformerForPrediction in a distributed environment based on the following notebook: https://huggingface.co/blog/time-series-transformers

If I run the notebook as is I get the following error immediately after the first loss is calculated: 
```
 Expected to have finished reduction in the prior iteration before starting a new one. This error indicates that your module has parameters that were not used in producing loss. You can enable unused parameter detection by passing the keyword argument `find_unused_parameters=True` to `torch.nn.parallel.DistributedDataParallel`, and by 
making sure all `forward` function outputs participate in calculating loss. 
If you already have done the above, then the distributed data parallel module wasn't able to locate the output tensors in the return value of your module's `forward` function. Please include the loss function and the structure of the return value of `forward` of your module when reporting this issue (e.g. list, dict, iterable).
Parameter indices which did not receive grad for rank 0: 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146
 In addition, you can set the environment variable TORCH_DISTRIBUTED_DEBUG to either INFO or DETAIL to print out information about which particular parameters did not receive gradient on this rank as part of this error
```

If I modify the accelerator in the following way:
```
ddp_kwargs = DistributedDataParallelKwargs(find_unused_parameters=True)
accelerator = Accelerator(kwargs_handlers=[ddp_kwargs])
```
It will usually run for many iterations but it randomly fails with an error message similar to above.

The code appears to run fine in cpu mode or on GPU in a non-distributed environment.

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Run the following notebook in a distributed GPU environment: https://huggingface.co/blog/time-series-transformers

### Expected behavior

I would expect the model to train and perform `.backward(loss)` without error.

Comments:
cc @kashif 

thanks @maeschbacher having a look

I don't think @kashif will have time to look at this, so marking as Good Difficult issue. 

Hey @maeschbacher, if you're still experiencing the issue, one potential fix is to include `static_graph=False` within `DistributedDataParallelKwargs`. This is because the graph may not be static even when you're using `DistributedDataParallelKwargs(find_unused_parameters=True)`. Give it a try and let me know if it works for you. 

Best regards!
Fix: failing CI after #30568
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes a failing test for Fuyu model after #30568. The solution is to change the corresponding config's vocab size, depending on if it's VLM or LLM. But the old way of setting config's vocab_size is not removed yet, since it's planned to be removed in v4.42 as [commented in Llava configuration.py](https://github.com/huggingface/transformers/blob/d2feb545918a325a3a8373db706ef6fe4ec5776f/src/transformers/models/llava/configuration_llava.py#L144) files

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30599). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
assisted_decoding does not correctly update static KV cache
Author: learning-chip
Pull request: no
Labels: name: Generation

name: Cache

Inside `assisted_decoding`, KV cache is cropped to the actual accepted tokens via `_crop_past_key_values`:

```python
            new_cache_size = new_cur_len - 1
            outputs.past_key_values = _crop_past_key_values(self, outputs.past_key_values, new_cache_size)
```
https://github.com/huggingface/transformers/blob/v4.39.3/src/transformers/generation/utils.py#L4576-L4578

However, `_crop_past_key_values` dynamically creates a new KV cache with shorter length:

```python
        for idx in range(len(past_key_values)):
            new_past.append(
                (
                    past_key_values[idx][0][:, :, :maximum_length, :],
                    past_key_values[idx][1][:, :, :maximum_length, :],
                )
            )
        past_key_values = tuple(new_past)
```
https://github.com/huggingface/transformers/blob/v4.39.3/src/transformers/generation/candidate_generator.py#L368-L375

With `StaticCache`, should use in-place write instead.

Comments:
@gante 

Because `cache_position` is updated accordingly, maybe no need to crop `past_key_values` at all? Next step the newly computed KVs (of speculative length) will be written into cache, overwriting the old values.

```python
                candidate_kwargs["cache_position"] = torch.cat(
                    (
                        candidate_kwargs["cache_position"],
                        torch.arange(cur_len, cur_len + candidate_length, device=input_ids.device, dtype=torch.long),
                    ),
                    dim=0,
                )
```
https://github.com/huggingface/transformers/blob/v4.39.3/src/transformers/generation/utils.py#L4505-L4511

@learning-chip currently assisted generation does not work for Static Cache if some candidate tokens are discarded. Ss you have mentioned past kv cache does not need to be cropped, but `cache_position` needs to be updated according to number of accepted candidate tokens. 

There is a [PR](https://github.com/huggingface/transformers/pull/30068) for fixing `cache position` in assisted generation, should be working after that  : )

I will close the issue as the PR was merged. To get the assisted generation working with static cache install the latest version from `main` with `!pip install --upgrade git+https://github.com/huggingface/transformers.git`

Feel free to open again if the problem persists 🤗
Fix for Neuron
Author: michaelbenayoun
Pull request: yes

# What does this PR do?

This PR fixes things for them to run on Trainium instances.

It fixes:

- The use of Ellipsis[ is not well supported](https://awsdocs-neuron.readthedocs-hosted.com/en/latest/general/appnotes/torch-neuronx/introducing-pytorch-2-1.html?highlight=ellipsis#incorrect-device-assignment-when-using-ellipsis) in the latest Neuron SDK
- The way metadata is computed during `symbolic_trace`. Before no metadata was traced when a user would define its custom leaf modules, it should be working now.
- The way the new `transformers.cache_utils.Cache` classes are handled. They can now be symbolically traced.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30259). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@michaelbenayoun There were a bunch of fp16 issues we had to solve last week because of the new pytorch release. Could you try rebasing on main? This _should_ resolve the currently failing tests

Some tests are failing because of the PR itself. Working on that today.

@amyeroberts about [this comment](https://github.com/huggingface/transformers/pull/30259#discussion_r1584740792), you can check my answer to @ArthurZucker .

To give a little bit more context, I'm doing this because the way it is currently implemented produces a compiler error on AWS Trainium / Inferentia devices, preventing us to use these models with the latest Transformers version.
Fix memory leak with CTC training script on Chinese languages
Author: lucky-bai
Pull request: yes

This PR fixes a memory leak that occurs during the evaluation phase with the CTC training script when attempting to train a CTC model on a language with a very large vocabulary size. I ran into this issue while training for Chinese, but it may affect other languages as well. The problem arises because all the logits are accumulated on the GPU when running the evaluation, even though it is batched, so all the logits for the entire evaluation set are accumulated in the GPU. This is not too much of a problem for most languages, but it quickly runs out of CUDA memory when the vocabulary size is as large as it is in the case for Chinese.

Some community discussion of this problem here: https://discuss.huggingface.co/t/cuda-out-of-memory-when-using-trainer-with-compute-metrics/2941

The bug may be reproduced using this command:

```
python run_speech_recognition_ctc.py \
	--dataset_name="mozilla-foundation/common_voice_11_0" \
        --model_name_or_path="facebook/wav2vec2-xls-r-300m" \
	--tokenizer_name_or_path="jonatasgrosman/wav2vec2-large-xlsr-53-chinese-zh-cn" \
	--dataset_config_name="yue" \
	--output_dir="./wav2vec2-common_voice-tr-demo" \
	--overwrite_output_dir \
	--num_train_epochs="15" \
	--per_device_train_batch_size="16" \
	--gradient_accumulation_steps="2" \
	--learning_rate="3e-4" \
	--warmup_steps="500" \
	--text_column_name="sentence" \
	--length_column_name="input_length" \
	--save_steps="400" \
	--eval_steps="100" \
	--layerdrop="0.0" \
	--save_total_limit="3" \
	--freeze_feature_encoder \
	--gradient_checkpointing \
	--chars_to_ignore , ? . ! - \; \: \" “ % ‘ ” � \
	--fp16 \
	--group_by_length \
	--push_to_hub \
	--do_eval 
```

cc @sanchit-gandhi 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30358). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix copies for DBRX - neuron fix
Author: amyeroberts
Pull request: yes

# What does this PR do?

Fixes quality checks on main failing after the merge of https://github.com/huggingface/transformers/pull/30259

Comments:
cc @ArthurZucker for reference for patch release 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30610). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Use `contiguous()` in clip checkpoint conversion script
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix #30037 

See https://github.com/huggingface/transformers/issues/30037#issuecomment-2069998788

@Narsil I fixed the issue, but maybe you want to share some insights/information.



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30613). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Issue with `output_router_logits` Parameter Not Being Passed Correctly in `SwitchTransformersForConditionalGeneration`
Author: lausannel
Pull request: no

### System Info

My environment is as follows:

```
- `transformers` version: 4.38.2
- Platform: Linux-6.5.0-27-generic-x86_64-with-glibc2.35
- Python version: 3.11.4
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.26.1
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```


### Who can help?

@ArthurZucker

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

When using SwitchTransformersForConditionalGeneration, I found that the output_router_logits parameter is not being passed correctly. Here is the reproducible code:

```python
from transformers import (
    AutoTokenizer,
    AutoConfig,
    SwitchTransformersForConditionalGeneration,
)

# define a decorator to measure the parameter passing
def forward_decorator(orig_forward):
    def check_forward(cls, *args, **kwargs):
        # print output_router_logits in kwargs
        output_router_logits = kwargs.get("output_router_logits", None)
        if output_router_logits is not None:
            print(f"output_router_logits: {output_router_logits}")
        else:
            print("output_router_logits is not in kwargs")
        result = orig_forward(cls, *args, **kwargs)
        return result
    return check_forward

SwitchTransformersForConditionalGeneration._old_forward = SwitchTransformersForConditionalGeneration.forward
SwitchTransformersForConditionalGeneration.forward = forward_decorator(SwitchTransformersForConditionalGeneration.forward)

checkpoint = "google/switch-base-8"
config = AutoConfig.from_pretrained(checkpoint)
tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = SwitchTransformersForConditionalGeneration.from_pretrained(checkpoint, config=config)

input_text = "Hello, my name is Susan. How are you?"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids

output = model.generate(
    input_ids,
    do_sample=True,
    output_router_logits=False,
)

print(tokenizer.decode(output[0], skip_special_tokens=True))
```



### Expected behavior

This code produces the following output:

```
output_router_logits is not in kwargs # Expected: output_router_logits: False
....
```

I suspect that this issue may be related to the prepare_inputs_for_generation function of the SwitchTransformersForConditionalGeneration. I plan to submit a PR to address this issue. Could you please review it once it's available? Thank you!

Comments:
phi3 chat_template does not support system role
Author: amitportnoy
Pull request: yes

# What does this PR do?

Removes misleading example which uses the system role

Phi3 chat_template does not use that role.
See https://huggingface.co/microsoft/Phi-3-mini-4k-instruct/blob/main/tokenizer_config.json#L119

- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@stevhliu @MKhalusova @ArthurZucker @younesbelkada


Comments:
> LGTM let's just make sure the CI is green 😉

The test that fails doesn't make sense to me, as the model's output is not deterministic.

The expected one should be (according to the CI failure trace):
```bash
    

 
    Can you provide ways to eat combinations of bananas and dragonfruits?
 
    
 
    Certainly! Bananas and dragonfruits can be combined in various delicious ways. Here are some ideas for eating combinations of bananas and
```

(tests passed)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30606). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Docs: add missing `StoppingCriteria` autodocs
Author: gante
Pull request: yes

# What does this PR do?

They were not in the main `__init__`, so `make fixup` didn't complain :)

cc @zucchini-nlp @Rocketknight1 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30617). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Generate: fix `SinkCache` on Llama models
Author: gante
Pull request: yes

# What does this PR do?

`SinkCache` has been broken on Llama and Llama-based models since we released the static cache update (v4.38). Now that we are happy with the state of the static cache (#30476), we can move on to fix what we broke along the way 🤗 

In a nutshell, the static cache rework changed the `sin` and `cos` tensors passed around, from the full set of values for all possible positions (up to `config. max_position_embeddings`) to the values used in a forward pass alone. This is a non-negotiable change to achieve top compiled performance.

However, `SinkCache` needs access to the whole `sin` and `cos` tensors, and they are not trivial to compute from scratch in the cache instance (it would need access to the RoPE class, creating a cyclical dependency). The `SinkCache` instance was changed to hold a cache [meta cache 🤯 ] of `sin` and `cos` as it sees them, rebuilding the full tensor internally. Having the full tensor rebuilt, it can operate as expected.

`tests/test_cache_utils.py::CacheIntegrationTest::test_sink_cache_hard` is fixed as a result of this PR. All other sink cache tests were passing (they were not using llama 👼 )

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30581). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Mistral&Mixtral]Add sliding window for sdpa
Author: ehuaa
Pull request: yes
Labels: name: SDPA

@ArthurZucker  Arthur has reviewed before, but my git changes log is weird, so i open a new pr instead. I uploaded a new test for slidingwindow flash vs sdpa for checking.

Superseeds #29220

Comments:
@fxmarty if you want to take over in a new PR, this is fairly important IMO

This PL will solve https://github.com/huggingface/transformers/issues/28980

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

closing as https://github.com/huggingface/transformers/pull/30127 was merged and takes inspiration from this PR
Remove pipelines, chatformatters, templates etc  --> Replace with simple generator function / manual string interpolation ---> Just have one standardized way for building datasets and running inference 
Author: bdytx5
Pull request: no
Labels: name: Chat Template

### Feature request

https://huggingface.co/docs/transformers/main/en/chat_templating

Theres a lot of complexity here. I dont think that it needs to be as complicated as it is. We can just look at what special tokens are being used for the model, and create a dataset based on what we need.

### Motivation

Its hard to tell whats actually being passed to the model, as there are several different pipeline and generators, its extremely difficult to know if your dataset is actually being formatted correctly 

### Your contribution

https://github.com/bdytx5/llama3_finetune/blob/main/tr.py

This is my best simplification. There's no need for chat templates or anything like that, as it's really just simple string interpolation. Trying to abstract this just creates more complexity and confusion. 

Comments:
Hi @bdytx5, thanks for opening this feature request! 

Pinging @Rocketknight1 as he's the master of the chat templates. 

I think this simplification might be a little too simplified, since it doesn't seem to support anything like system prompting, multi-turn conversations, or any model other than LLaMA-3. It also doesn't support e.g. control over whether we append a generation prompt.

We did consider the solution of just using string formatting rather than a templating language, but what we found was that many models have templates too complex for a universal string formatting solution to work. In particular, here are some tricky ones:

- LLaMA-2 [injects the system message](https://huggingface.co/meta-llama/Llama-2-7b-chat-hf/blob/main/tokenizer_config.json), if present, inside the first user message.
- Command-R has [three different templates](https://huggingface.co/CohereForAI/c4ai-command-r-plus/blob/main/tokenizer_config.json#L304), for basic generation, tool use and RAG generation. Arguments for the latter two can be optional or nested.

In addition, any replacement solution would need, at a minimum, the following properties:
- Has to support all known existing model chat formats (including the two listed above, but also many others)
- Has to work with a single universal Python function in the `transformers` codebase - no code in the model repos, which includes f-strings, as these allow arbitrary execution!
- Ideally not Python-specific, since Rust libraries like TGI are also handling chat templates via `minijinja`, or `huggingface/jinja` for JS.

I think you'll find that if you expand your solution to cover all of these possible cases, you'll end up reinventing chat templates, or something that looks a lot like them!

Good-naturedly closing this for now, but feel free to reopen it if you want to propose a more complete replacement for chat templating.