noamaanMulla-03/transformers-issues huggingface dataset:
Generate: handle `cache_position` update in `generate`
Author: gante
Pull request: yes

# What does this PR do?

Updates `cache_position` in `generate`, and makes it the primary source for the input position in the models that support them, `llama` and `gemma` (as opposed to relying on `past_key_values.seen_tokens`).

The PR also adds the following related changes:
1. `StaticCache` now supports `get_seq_length()`. This was drawn from #29221, and is needed for `.prepare_inputs_for_generation()` retrocompatibility;
2. The `seen_tokens` attribute enters a deprecation cycle, as it is redundant with `cache_positions` (and doesn't work with compilation).

This PR is drawn from the diff in #29374, i.e. it is a requirement for `generate` compilation with `fullgraph=True` 🙌 

👉 Llama, Gemma, and Cache slow tests ran, no new failures
👉 FWD compilation benchmarks ran, no throughput change

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29467). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

(rebasing and reruning tests, just in case 🙃 )
Allow apply_chat_template to pass kwargs to the template and support a dict of templates
Author: Rocketknight1
Pull request: yes

As the title suggests, this is a simple PR that allows two new features:

- `kwargs` can be passed through `apply_chat_template` to the template renderer.
- Models can have a dict of multiple chat templates, which can be accessed by passing their name to `apply_chat_template` (this is used by the new Command-R model)

This PR is very slightly breaking (we used to pass kwargs to the tokenizer), but I don't think this was commonly used, and I think the new usage is more intuitive. Users can still pass kwargs through the method to the tokenizer with the `tokenizer_kwargs` argument.

Other than that, it should have no effect on existing users/templates!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29658). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Inaccurate code example within inline code-documentation
Author: thariq-nugrohotomo
Pull request: no
Labels: name: Documentation

name: Good First Issue

name: bug

I believe this code example contains inaccurate information:

https://github.com/huggingface/transformers/blob/211ad4c9cc1c0882c4a22eaca7b4d7d1e2f264b3/src/transformers/models/vision_encoder_decoder/modeling_vision_encoder_decoder.py#L550C15-L550C15

```python
model.config.decoder_start_token_id = processor.tokenizer.cls_token_id
```

TrOCR decoder_start_token should be `eos` instead of `cls` or `bos`.

Using the pretrained model, when I pass `cls` or `bos` as the initial decoder token, the output (first decoded token) rarely get correct. But once I try to use `eos`, the output is correct, or at least similar with the output returned by `model.generate()`.

In the official code from Microsoft, they will fallback to `eos` if the token is not specified https://github.com/microsoft/unilm/blob/6f60612e7cc86a2a1ae85c47231507a587ab4e01/trocr/generator.py#L84

Code excerpt to manually see the first decoded token:

```python
decoder_start_token_id = processor.tokenizer.eos_token_id
x = model(pixel_values, torch.tensor([[decoder_start_token_id]]))
x = x.logits
x = torch.argmax(x, -1)
print(processor.tokenizer.batch_decode(x))
```

Switch `eos_token_id` to `cls_token_id` or `bos_token_id` to observe the incorrect output.

Comments:
Hi @thariq-nugrohotomo, thanks for raising this issue! 

Indeed, it seems that the behaviour in the original implementation is different. Would you like to open a PR updating the code examples in modeling_vision_encoder_decoder.py and modeling_trocr.py? This way you get the github contribution. 

I would like to fix the bug, is it only this file that needs to be changed or are there other places that require changes too ? @amyeroberts 

@MysteryManav I would suggest searching for TROCR checkpoints being used in other examples in the codebase in case that should also be updated. Just updating the vision encoder decoder example is OK though
Inaccurate code example within inline code-documentation
Author: MysteryManav
Pull request: yes

Changes the inline code-documentation for better and accurate example

Fixes #27142 


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
@amyeroberts i have changed the inline-documentation in the following files:
1. modeling_vision_encoder_decoder.py
2. modeling_trocr.py

I am not sure if the change in modeling_git.py would be helpful or not, so please do look into that, and if there are other files that require the same change, if i am correct, most of the files that start with "modeling_" have the documentation, should i apply the changes to those too ? I have not changed other files, since it seems that might caues some functionalities to break

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29661). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Alternative way of creating decoder_attention_mask to avoid torch warning about new_tensor()
Author: kmartiny
Pull request: yes


# What does this PR do?

Replace the method of creating `decoder_attention_mask` in `modeling_encoder_decoder.py` to avoid a pytorch warning. Masks were created as 
```python
decoder_attention_mask = torch.tensor(decoder_input_ids != self.config.pad_token_id)
```
but this results in a torch warning:
> transformers/models/encoder_decoder/modeling_encoder_decoder.py:620: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than tensor.new_tensor(sourceTensor).
  decoder_attention_mask = decoder_input_ids.new_tensor(decoder_input_ids != self.config.pad_token_id)

Creating the masks through
```python
decoder_attention_mask = torch.not_equal(decoder_input_ids, self.config.pad_token_id)
```
is equivalent without producing warnings.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[tests] extend `NOT_DEVICE_TESTS`
Author: faaany
Pull request: yes

## What does this PR do?
As a guy working on XPU, I found PR #29213 extremely useful, but I discovered more test cases that will not be tested on accelerators. 

The only concern I have is the `repository-skip`, e.g. currently I find all files under `repo_utils`, `tools` and `utils` repositories belong to `NOT_DEVICE_TESTS`, because these test files don't have the keyword `torch_device` or `gpu`. But this doesn't guarantee that people will add DEVICE-relevant tests to these repositories in the future. What are your thoughts on this? 

@amyeroberts @fxmarty @ydshieh 

Comments:
Need to DeformableDetrModel support fp16
Author: DonggeunYu
Pull request: no

### Feature request

Need to DeformableDetrModel using fp16.

~~~
from transformers import AutoImageProcessor
from trf.models import DeformableDetrModel
from PIL import Image
import torch
import requests

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

image_processor = AutoImageProcessor.from_pretrained("SenseTime/deformable-detr")
model = DeformableDetrModel.from_pretrained("SenseTime/deformable-detr")
model = model.cuda().half()

inputs = image_processor(images=image, return_tensors="pt")
inputs["pixel_values"] = inputs["pixel_values"].cuda().half()
inputs["pixel_mask"] = inputs["pixel_mask"].cuda()
outputs = model(**inputs)

print(outputs)
~~~

Output:
~~~
File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/deformable_detr/modeling_deformable_detr.py", line 699, in forward
    output = MultiScaleDeformableAttentionFunction.apply(
  File "/usr/local/lib/python3.10/dist-packages/torch/autograd/function.py", line 539, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/deformable_detr/modeling_deformable_detr.py", line 81, in forward
    output = MultiScaleDeformableAttention.ms_deform_attn_forward(
RuntimeError: "ms_deform_attn_forward_cuda" not implemented for 'Half'
~~~

### Motivation

FP16 for speed-up training.

### Your contribution

Yes. I'm planning to submit a PR.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Pytorch not detected in official "editable" nightly
Author: bhack
Pull request: no

### System Info

Pytorch is not detecting in the offical editable nighlty conda env:
https://github.com/pytorch/pytorch/blob/main/CONTRIBUTING.md#nightly-checkout--pull
```python
  File "/opt/conda/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1325, in __getattribute__
    requires_backends(cls, cls._backends)
  File "/opt/conda/lib/python3.11/site-packages/transformers/utils/import_utils.py", line 1313, in requires_backends
    raise ImportError("".join(failed))
ImportError: 
BertForMaskedLM requires the PyTorch library but it was not found in your environment. Checkout the instructions on the
installation page: https://pytorch.org/get-started/locally/ and follow the ones that match your environment.
Please note that you may need to restart your runtime after installation.
```

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://github.com/pytorch/pytorch/issues/119740#issuecomment-1941470322

### Expected behavior

pytorch need to be detected also in this official dev-env

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

We have a PR
Trainer: fail early in the presence of an unsavable `generation_config`
Author: gante
Pull request: yes

# What does this PR do?

Fixes #29665 

We don't allow saving bad generation config files (i.e. ones whose validation has warnings or exceptions). This check was only done at save time, which happens AFTER training. This PR introduces the same check in the trainer's `__init__`, to fail early (and before spending time/$)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29675). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Misleading warning message about pad_token_id when passing tokenizer instance to pipeline
Author: tilmanbeck
Pull request: no

### System Info

Python=3.11.5
Transformers= '4.37.2'

In my setup I am initializing a tokenizer and want to pass it to the pipeline. My expectation is that if I set the `pad_token_id` directly on tokenizer instance, the pipeline should not print the warning `Setting pad_token_id to eos_token_id:50256 for open-end generation.` If I pass the `pad_token_id` directly to the pipeline's `__call__`, the warning is not printed. However, I would rather like to set the `pad_token_id` directly rather than having to think about everytime I use the `__call__` method of the pipeline.

Alternatively, I would like to add it as a parameter to the pipeline instantiation, but I think passing tokenizer parameters in the pipeline generator is currently not envisioned, as discussed in #12039 #24707 #22995 

@Narsil 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Here is a minimal code example to demonstrate the difference:
```
from transformers import AutoTokenizer, pipeline, AutoModelForCausalLM, Conversation

msg = 'The capital of France '
modelname = 'microsoft/DialoGPT-small'

# hand over model & tokenizer instantiations to pipeline
model = AutoModelForCausalLM.from_pretrained(modelname)
tokenizer = AutoTokenizer.from_pretrained(modelname)
tokenizer.pad_token_id = tokenizer.eos_token_id

chatbot = pipeline(task='conversational', model=model, tokenizer=tokenizer, framework='pt')
messages = ([{"role": "system", "content": 'You are a helpful assistant'}, {"role": "user", "content": msg}])
response = chatbot(Conversation(messages=messages))
```
This prints the warning message
`Setting 'pad_token_id' to 'eos_token_id':50256 for open-end generation.`

The following code does not:
```
from transformers import AutoTokenizer, pipeline, AutoModelForCausalLM, Conversation

msg = 'The capital of France '
modelname = 'microsoft/DialoGPT-small'

chatbot = pipeline(task='conversational', model=modelname, framework='pt')
messages = ([{"role": "system", "content": 'You are a helpful assistant'}, {"role": "user", "content": msg}])
response = chatbot(Conversation(messages=messages), pad_token_id=chatbot.tokenizer.eos_token_id)
```

### Expected behavior

I would expect the first code snippet not to print the warning as the `pad_token_id` is directly set on the tokenizer instance

Comments:
cc @Rocketknight1 

This issue and #29379 feel a bit like generation issues to me - @gante do you see any obvious solutions? If not, don't worry, I'll investigate.

Hi @tilmanbeck 👋 

The root issue is that the `tokenizer` and the `model` are two separate objects, so `pad_token_id` needs to be set in both. In `pipeline`, IMO `model` should inherit `pad_token_id` from `tokenizer` when it is not set there. Opening a PR to fix that :)
Pipeline: use tokenizer pad token at generation time if the model pad token is unset.
Author: gante
Pull request: yes

# What does this PR do?

Fixes #29378 

The tagged issue describes the problem, the title describes the fix :D

Example of a script that no longer emits a warning, after this PR:
```py
from transformers import AutoTokenizer, AutoModelForCausalLM, pipeline

model = AutoModelForCausalLM.from_pretrained("openai-community/gpt2")
tokenizer = AutoTokenizer.from_pretrained("openai-community/gpt2")
tokenizer.pad_token_id = tokenizer.eos_token_id

llm = pipeline(task='text-generation', model=model, tokenizer=tokenizer, framework='pt')
response = llm('The capital of France ')
```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29614). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts ready for a re-review :)
Cohere Model Release
Author: saurabhdash2512
Pull request: yes

Cohere Model Release

# What does this PR do?

This PR adds the Cohere Model Family with the release of the [Command-R model weights](https://huggingface.co/CohereForAI/c4ai-command-r-v01)

More information about the model can be found [here](https://txt.cohere.com/command-r/)


## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@LysandreJik


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29622). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @saurabhdash2512, this looks really good! We've just made some updates to the `transformers` library [here](https://github.com/huggingface/transformers/pull/29658) to support some of the new chat template features this model uses - if possible, can you rebase your PR so we can use them in this port? I'll make a PR to your branch to add support for the features once you rebase!

(I realize you've already rebased quite recently - I'm sorry about this!)

> Hi @saurabhdash2512, this looks really good! We've just made some updates to the `transformers` library [here](https://github.com/huggingface/transformers/pull/29658) to support some of the new chat template features this model uses - if possible, can you rebase your PR so we can use them in this port? I'll make a PR to your branch to add support for the features once you rebase!
> 
> (I realize you've already rebased quite recently - I'm sorry about this!)

@Rocketknight1 Done! Sync'ed with main.
Fix AutoformerForPrediction example code
Author: m-torhan
Pull request: yes

# What does this PR do?

The pretrained Autoformer config [config.json](https://huggingface.co/huggingface/autoformer-tourism-monthly/blob/main/config.json#L55) suggests that this model was trained without using static real features. However, in the sample code the model is called with it which causes exception to be raised as mentioned in #29565. This PR removes that parameter from model call and fixes the issue.

Fixes #29565

## Who can review?
@kashif

Comments:
wonder if it makes sense to just put a comment on those? I still wanted folks to know how to call this with these features if they exist?

Added a tip about static real features usage

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29639). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix wrong condition used in `filter_models`
Author: ydshieh
Pull request: yes

# What does this PR do?

My previous #28816 used a stupid wrong condition.

The correct one should be : we don't want to filter when we are on `main` branch.

I am so bad at boolean op, sorry

cc @gante 


Comments:
Happens to the best of us, @ydshieh! Thank you for fixing 💛 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29673). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Rename `glue` to `nyu-mll/glue`
Author: lhoestq
Pull request: yes

`glue` has been moved to `nyu-mll/glue`

Note that we redirect anyway in `load_dataset`, this change is just to include the dataset owners name as we're moving all the datasets without namespaces to their owner/creator accounts

Comments:
I just renamed it in two other places: the tf one and the no_trainer one :)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29679). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Quantization] Quanto quantizer
Author: SunMarc
Pull request: yes

# What does this PR do ? 
This PR adds the quantization methods from quanto library. We will support inference + model quantization if the user perform weights only quantization since we don't require a calibration dataset. 

TODO:
- [x] docs 
- [x] tests
- [x] Guard against saving now since it is a bit complicated

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29023). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @dacorvo 

Quick update, all tests are passing with the exception of safetensors tests. I've also implemented the quantization on the spot just like we do for bnb quantization. However, if the user wants to use cpu/disk offload, he will need to install the main branch of accelerate for now because of this [PR](https://github.com/huggingface/accelerate/pull/2481). 

I've fixed the issues you mentioned ! LMK this looks good and I'll ask a final review from a core maintainer ! 

LGTM, mostly concerned with quanto specific function addition, that should leave in quanto.py not in the modeling (as much as possible of course). 

Note to the core mainteners and especially @ArthurZucker . I reverted a lot of changes I did for the serialization. I decided to postpone the serialization feature by setting the `is_serializable` property to `False` in this PR. 

I will do a follow up PR for the serialization since the [PR](https://github.com/huggingface/quanto/pull/116) to make it compatible with safetensors or `weight_only` saving in quanto needs lots of changes in the way we load + save models in transformers. 

Without serialization, this PR should still be good to merge !