noamaanMulla-03/transformers-issues huggingface dataset:
Move `eos_token_id` to stopping criteria
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

This PR is a small step for `torch.compile` and `generate` compatibility. It moves `EOS` token to stopping criteria, so now we can loop `while stopping_criteria` and get rid of extra checks on `EOS` at the end of each generate method. 

All the generate tests, including slow are passing.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [X] Did you write any new necessary tests?

## Who can review?

@gante 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29459). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Ah, the pipeline tests are ignored by default (just like slow tests). You need to add `RUN_PIPELINE_TESTS=1` to run them :)

Done for all comments. I checked that users have to possibility of messing up when calling the these methods directly and ran again all tests (+slow, +pipeline)

@zucchini-nlp needs a rebase with `main` and it should be ready to be merged 

(@ArthurZucker I'm assuming this is ready to be merged, since all comments were addressed. I'm deciding to merge since this unblocks me on the torch.compile front, let us know if you like some post-merge changes :) )

@gante Woah, hold on before merging! You still need a core maintainers approval: even if the comments have been addressed it's important to make sure the amendments are approved

@zucchini-nlp this PR needs to be rebased with `main` to fix CI :)

@ArthurZucker ping for approval, if all tasks are complete :)

Rebased `main` and ran all tests in generation again, including slow, ran a few generation models' tests. Everything is passing, can be merged now. @gante 

No idea why the examples test is failing, does not have anything to do with this PR.

Just a heads up this commit breaks transformers generation on Apple Silicon as isin is not implemented for the MPS backend
Pipeline: fix `pad_token_id` again
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Related to PR #30285. The earlier PR caused audio-to-text pipelines to fail in daily doctest CI, because they do not accept pad-token-id in kwargs. This should be a better solution, to set pad token directly in model's config

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30338). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Yes, all tests in pipeline + doctest are running green on my end mostly. Some tests that are failing are not related to the fix, maybe hardware issues

CI passing now :) it was an temporary issue on the ubuntu package servers, it seems
Add recurrent gemma
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Adds support for recurrent gemma

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30143). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Just making sure the slow generations are correct, and merging!
parallel job limit for doctest
Author: ydshieh
Pull request: yes

# What does this PR do?

@glegendre01 mentioned we limit to set a limit for doctest workflow file, otherwise it will use all runners in the runner group.

Comments:
it seems better. let's try try if it works.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30342). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Deprecate default chat templates
Author: Rocketknight1
Pull request: yes

Before we had chat templates, models had class-specific chat handling hardcoded into their `tokenization.py` files. In order to preserve backward compatibility, this class-level code was replaced with "default" chat templates. However, this has been a source of problems, because the default chat templates are often invalid for a given model, and because the default chat templates are being applied even to base models that have not been finetuned for chat or instruction following.

The whole point of chat templates was to be explicit, and to avoid silent errors and failures caused by incorrect templates being invisibly applied. Therefore, now that chat templates are widely adopted, I think the best solution is to deprecate default chat templates and remove them in the near future.

Default chat templates already raised warnings to indicate that they were error-prone; this PR makes the warnings far more explicit and sets a clear deprecation timeline. It also updates the chat template documentation to indicate that default chat templates are going away.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30346). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Restore casting of masked_spec_embed
Author: ylacombe
Pull request: yes

# What does this PR do?

Restore a casting removed by #30310, that caused training examples to fail.

cc @ydshieh ! 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30336). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update unwrap from accelerate
Author: SunMarc
Pull request: yes

# What does this PR do ? 
This PR update the unwrap function to use the one in accelerate instead. 
Fixes issue from @abhishekkrthakur 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29933). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Got word from tf-boi that generally `modeling_utils.py` is PyTorch only, so any code that calls it requires pytorch, and in-vein with how Accelerate is with transformers requirement wise, `accelerate`. So I propose that we do move forward with fully ripping it out and relying on accelerate, modifying it's implementation. 

@amyeroberts @ArthurZucker what do you say? :) 

Hi @muellerzr @SunMarc , does this PR will merged? We need this.

I will finish this PR asap @zorrofox ! @ArthurZucker , do you want to switch back to `unwrap_model` in the trainer files also ? It makes sense to use `self.accelerator.unwrap_model(model)` since we initialized `Accelerator` there. I've updated the `unwrap_model` function, LMK what you think.
Do not remove half seq length in generation tests
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Generation tests divided input ids into half, feeding only half-seq-length for generation. But this strategy may cause difficulties for multimodal models, which have dependency between images count and input ids. For ex, Llava models include special image tokens in input ids, and removing half-seq-length might result in failing test. 

This is part of a work to add GenerationTesterMixin in multimodal models, which are not covered with tests now.

This PR: 
- removes dividing seq length into half
- replaces "max length" with "max new tokens" for ease
- gets rid of "min length" as we already have "config.eos_token=None" everywhere ensuring that exactly "max length" will be generated
  

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@gante 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30016). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@zucchini-nlp Can you trigger all tests, to double-check? (i.e. commit message = "[test_all] ...")

All tests are passing, except for the attached PR and another which seemingly failed to download files.
Fix config + attn_implementation in AutoModelForCausalLM.from_pretrained
Author: hiyouga
Pull request: yes

# What does this PR do?

Fixes #30298 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@ArthurZucker and @younesbelkada 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30299). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> This looks very reasonable, thank you for the fix. Ideally, could you add a test for this? For example, in `tests/test_modeling_utils.py`'s `TestAttentionImplementation`.

Ok I'll do it

@fxmarty I have done, could you review it again?

@hiyouga something is wrong with the CI could you try merging main?

> @hiyouga something is wrong with the CI could you try merging main?

Done
Attention implementation cannot work together with config in AutoModel
Author: hiyouga
Pull request: no

### System Info

- `transformers` version: 4.40.0.dev0
- Platform: Linux-5.15.0-100-generic-x86_64-with-glibc2.35
- Python version: 3.11.8
- Huggingface_hub version: 0.21.4
- Safetensors version: 0.4.2
- Accelerate version: 0.28.0
- PyTorch version (GPU?): 2.2.0+cu121 (True)

### Who can help?

@younesbelkada 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Similar to https://github.com/huggingface/transformers/issues/28038

We want to pass a model config to `from_pretrained` with an `attn_implementation` parameter. The attention type cannot be faithful to the one in the `attn_implementation`

```python
from transformers import AutoConfig, AutoModelForCausalLM
model_name = "meta-llama/Llama-2-7b-hf"
config = AutoConfig.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, config=config, attn_implementation="eager")
print(model.config._attn_implementation)
# sdpa
```


### Expected behavior

`_attn_implementation` should be `eager`


Comments:
Given the logic below, we cannot enforce the model to use eager attention, since `config._attn_implementation` falls back to eager when `config._attn_implementation_internal` is None [1]. Hence, the if condition `config._attn_implementation != kwarg_attn_imp` cannot hold, and the `config._attn_implementation_internal` will be not affected, resulting a SDPA attention [2].

https://github.com/huggingface/transformers/blob/e4ea19b958c89d61e42461fac6ac8441787121f8/src/transformers/modeling_utils.py#L3138-L3150

I think we should use `config._attn_implementation_internal != kwarg_attn_imp` instead

1: 

https://github.com/huggingface/transformers/blob/e4ea19b958c89d61e42461fac6ac8441787121f8/src/transformers/configuration_utils.py#L406-L420

2: 
https://github.com/huggingface/transformers/blob/e4ea19b958c89d61e42461fac6ac8441787121f8/src/transformers/modeling_utils.py#L1461-L1466

cc @fxmarty
TF Swiftformer
Author: amyeroberts
Pull request: no
Labels: name: New model

### Model description

Add the TensorFlow port of the SwiftFormer model. See related issue: #22685 

To be done once the SwiftFormer model has been added: #22686 

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

Original repo: https://github.com/amshaker/swiftformer

Comments:
Hi! I would like to take on this as my first issue if possible. Is that okay?

@joaocmd Of course! Cool issue to start on 🤗 

If you haven't seen it already, there's a detailed guide in the docs on porting a model to TensorFlow: https://huggingface.co/docs/transformers/add_tensorflow_model

It's best to wait until the PyTorch implementation is merged in, which will be at least a day or two away. 

Sounds great @amyeroberts, thank you :)

I'll start looking into it once the PyTorch PR is merged.

Any news on this tf model? 

Hi @D-Roberts, I haven't started looking into it as the pytorch version has not yet been merged.

Hi @joaocmd , the PyTorch version of SwiftFormer is now merged so you can continue working the TensorFlow version of it.

Hi @shehanmunasinghe, I'm on it, thanks!

Hi @amyeroberts ,

May I proceed to work on this issue if it has not been sorted yet?

@sqali There is currently an open PR which is actively being worked on by @joaocmd: #23342 

Hi @amyeroberts ,

Is there any other issue that you are working on in which I can help you with?

@sqali Here's a page in the docs all about different ways to contribute and how to find issues to work on: https://huggingface.co/docs/transformers/main/contributing. I'd suggest looking at issues marked with the 'Good First Issues' tag and finding one which no-one is currently working on: https://github.com/huggingface/transformers/labels/Good%20First%20Issue
[Grounding DINO] Add resources
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds links to demo notebooks for Grounding DINO.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30232). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts feel free to approve and merge
Add model hiera cont
Author: Namangarg110
Pull request: yes

## What does this PR do?
Adds Hiera model from Meta as suggested in https://github.com/huggingface/transformers/issues/28993

GitHub Repo: https://github.com/facebookresearch/hiera/
arXiv: https://arxiv.org/abs/2306.00989

Model License recently changed to Apache

This PR is to merge the work done by @EduardoPach and me in #29945 and #29339 respectively.

c.c. @amyeroberts

Comments:
RuntimeError: Expected one of cpu, cuda, ipu, xpu, mkldnn, opengl, opencl, ideep, hip, ve, fpga, ort, xla, lazy, vulkan, mps, meta, hpu, mtia, privateuseone device type at start of device string: auto
Author: yumlevi
Pull request: no

### System Info

git latest
linux
h100 x 8 node

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

run llama3 example script

```python
import transformers
import torch

model_id = "meta-llama/Meta-Llama-3-70B-Instruct"

pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device="auto",
)

messages = [
    {"role": "system", "content": "You are a pirate chatbot who always responds in pirate speak!"},
    {"role": "user", "content": "Who are you?"},
]

prompt = pipeline.tokenizer.apply_chat_template(
        messages, 
        tokenize=False, 
        add_generation_prompt=True
)

terminators = [
    pipeline.tokenizer.eos_token_id,
    pipeline.tokenizer.convert_tokens_to_ids("
")
]

outputs = pipeline(
    prompt,
    max_new_tokens=256,
    eos_token_id=terminators,
    do_sample=True,
    temperature=0.6,
    top_p=0.9,
)
print(outputs[0]["generated_text"][len(prompt):])
``` 

### Expected behavior

work?, works when using mistral 22b with auto.

Comments:
cc @ArthurZucker 

Haven't tested but shouldn't it be:
```python
pipeline = transformers.pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device_map="auto",
)
```

`Device` usually requires the specific device type as stated in the error while `device_map` iirc uses accelerate to infer your device(s). There is also an inconsistency in the docs here when you compare the instruct and non-instruct variants (i.e. non-instruct suggests `device_map`). See the [source code](https://github.com/huggingface/transformers/blob/745bbfe4bb2b61491dedd56e1e8ee4af8ef1a9ec/src/transformers/pipelines/__init__.py#L674).

tested, worked fine with device_map='auto'
Flash Attention 2 for audio/musicgen
Author: yoinked-h
Pull request: no
Labels: name: Good Second Issue

name: Feature request

### Feature request

Add flash attention 2 to musicgen models (and or audio gen)

### Motivation

Musicgen is a *really* large model, and its very hard to run on consumer GPUs, adding flash attention could make the model much lighter and possibly faster.

### Your contribution

I can test on both WSL and windows 11 on a 3060

Comments:
cc @sanchit-gandhi @ylacombe 

Would indeed be cool having FA2 support for MusicGen! Since MusicGen copies the attention mechanism from Bart, you can copy the changes from this PR: https://github.com/huggingface/transformers/pull/27203

Would you like to have a go at this integration @yoinked-h? It should be quite a fun PR where you get quite big speed-ups for not too much additional effort!

I've tried to implement the Bart flash attention as best as i could, but i get some error (`cu_seqlens_k must have shape (batch_size + 1)`) from the module itself, will try tomorrow!

I would like to help if possible

Hey @yoinked-h and @staghado, as @sanchit-gandhi pointed it should be fun to add! 
Feel free to open a PR so that you can have our help on this

we can close this due to #29939

amazing!
show we add <image> token to the prompt if we use llava-hf from transformers?
Author: DefTruth
Pull request: no

### System Info

For the official llava (not the one in transformers), need to add an <image> token for inference:  
```python
inp = DEFAULT_IMAGE_TOKEN + '\n' + inp
```
show we add <image> token to the prompt if we use llava-hf from transformers?


### Who can help?

@amyeroberts @ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

none

### Expected behavior

none

Comments:
https://github.com/haotian-liu/LLaVA/blob/4e2277a060da264c4f21b364c867cc622c945874/llava/serve/cli.py#L83


Hi @DefTruth ! Yes, the special `<image>` token should be added to generate using an image. See [here](https://huggingface.co/docs/transformers/en/model_doc/llava#transformers.LlavaForConditionalGeneration.forward.example) for an example generation code 🤗 


thanks, i will try.
Model.generate encoder decoder
Author: Jim0530
Pull request: no

Instead of using model.generate()
How can I get the output of the decoder by taking the encoder hidden states and a start token to loop over the decoder model

cuz recently model.generate doesn’t fit my custom model.
Are there any examples or tutorial 
Thank you very much

Comments:
Hi,

I've answered your question on the forum: https://discuss.huggingface.co/t/generate-without-using-the-generate-method/11379

Please ask this question on our [forum](https://discuss.huggingface.co/) instead of here. We like to keep Github issues for bugs or feature requests.

Thanks!

> Hi,
> 
> I've answered your question on the forum: https://discuss.huggingface.co/t/generate-without-using-the-generate-method/11379
> 
> Please ask this question on our [forum](https://discuss.huggingface.co/) instead of here. We like to keep Github issues for bugs or feature requests.
> 
> Thanks!

from transformers import BlipForConditionalGeneration,AutoProcessor,AutoTokenizer
from PIL import Image
import requests
import torch
processor= AutoProcessor.from_pretrained("Salesforce/blip-image-captioning-base")
model= BlipForConditionalGeneration.from_pretrained("Salesforce/blip-image-captioning-base")
tokenizer = AutoTokenizer.from_pretrained("Salesforce/blip-image-captioning-base")

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
text = "A picture of"

decoder_input_ids = [model.decoder_input_ids]
print(decoder_input_ids)
inputs.input_ids = decoder_input_ids=torch.tensor([decoder_input_ids])
predicted_ids = []
for i in range(20):
    outputs = model(**inputs)
    logits = outputs.logits[:,i,:]
    print(outputs.logits.shape)
    predicted_id = logits.argmax(-1)
    predicted_ids.append(predicted_id.item())
    print(tokenizer.decode([predicted_id.squeeze()]))
    add predicted id to decoder_input_ids
    inputs.input_ids = torch.cat([inputs.input_ids, predicted_id.unsqueeze(0)], dim=1)


for this code for every image i found  same logits shape [1, 5, 30524] . and also every iteration the logits are  remain same. no change in logtis for changing the inputs.input_ids.  why i face this error? and how to solve this?

Hi, you should take the logits of the final token in the sequence, i.e. `logits = outputs.logits[:,-1,:]`.