noamaanMulla-03/transformers-issues huggingface dataset:
Confusing behavior when loading PEFT models with pipeline
Author: XEric7
Pull request: no
Labels: name: Feature request

### Feature request

Currently, when using transformers.pipeline to load a PEFT fine-tuned model (e.g., "ybelkada/opt-350m-lora"), the pipeline loads only the base model without applying the LoRA adapters. This behavior is misleading because users would expect to get the fine-tuned version of the model rather than just the base model.

For example:
```python
import transformers
pipeline = transformers.pipeline("text-generation", model="ybelkada/opt-350m-lora")
```
This code loads the base OPT model instead of the LoRA fine-tuned opt-350m-lora model.

I propose one of the following improvements:
1. Add a warning message when loading a PEFT fine-tuned model without applying LoRA adapters. For example:
    >  "Warning: You are loading a LoRA fine-tuned model, but the LoRA adapters have not been applied. Use PeftModel.from_pretrained() to correctly load the fine-tuned version."
2. Automatically detect and apply LoRA adapters when using pipeline, similar to how AutoModelForCausalLM.from_pretrained() works with PEFT models.


### Motivation

This issue is problematic because it creates a false assumption that users are working with the fine-tuned model when, in reality, they are only using the base model. Many users might not realize this and get incorrect results without knowing why. A clear warning or automatic application of LoRA would significantly improve user experience and reduce confusion.

### Your contribution

I can help test the implementation if needed. Let me know if there are any specific areas where contributions would be useful.

Comments:
Ah, that's a surprising bug! It must be specific to the pipeline loading code. Self-assigning.

Fix is open at #36480

> Fix is open at [#36480](https://github.com/huggingface/transformers/pull/36480)

Thanks a lot for the quick fix! I really appreciate the fast response and effort in resolving this issue. 🚀
Fix pipeline+peft interaction
Author: Rocketknight1
Pull request: yes

When you pass a PEFT adapter checkpoint to `pipeline()`, it actually loads the base model without the adapter. The reason is a bug in the `pipeline()` code, which has to look up the base model repo to get the config/architecture for the model, but accidentally totally clobbers the adapter path when it does.

Fixes #36473

Comments:
Ready for core maintainer review! cc @cyrilvallez @ArthurZucker 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36480). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
tokenizers.apply_chat_template with `continue_final_message=True` with trailing spaces in input
Author: chuyishang
Pull request: no
Labels: name: bug

name: Chat Template

### System Info

As title says, `tokenizers.apply_chat_template` fails with trailing spaces in input for Llama-3.1-Instruct.

If the last `assistant` message has a trailing space, such as
{'role': 'assistant', 'content': 'some text '}

and `continue_final_message` is True, it throws a "ValueError: substring not found"

This is because in the `apply_chat_template` function, there is a line
`rendered_chat = rendered_chat[: rendered_chat.rindex(final_message) + len(final_message)].rstrip()`

but `rendered_chat` ends with `"some text
"` while the `final_message` still has the trailing space: `"some text "`





### Who can help?

@ArthurZucker @itazap

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

See above

### Expected behavior

I expect it to be able to continue after the trailing space

Comments:
cc @Rocketknight1 

Hi @chuyishang, this should be fixed now! Please try updating to the latest version, or installing from `main` if that doesn't work.

@Rocketknight1 
There is a new issue introduced from this fix. 

When using multi-round few-shot such as 

```
[
    {"role": "user", "content": f"hi 0"},
    {"role": "assistant", "content": f"bye: 0"},
    {"role": "user", "content": f"hi 1"},
    {"role": "assistant", "content": f"bye: "}
]
```

The rendered chat with `tokenizer.apply_chat_template(messages, tokenize=False, continue_final_message=True, add_generation_prompt=False)` will actually completely remove the last round _if_ the tokenizer strips the text. Giving you 

```
<bos><start_of_turn>user
hi 0<end_of_turn>
<start_of_turn>model
bye: 
```

This is because the last round `"bye: "` is _only_ found in the penultimate round since the last round is now just `"bye:"` (no white space). 

I'm not sure what would be the best way to remedy this. Maybe `max(rindex(final_message), rindex(final_message.strip()))`?

@TK-21st yes, you're completely correct and I hadn't considered that! Working on a high-priority fix now

Hi @TK-21st, I've opened a PR to fix this at #36404. After investigation, the error you've found is very template-specific. Can you tell me what template you were using? I'll add a test using it to the `transformers` codebase so we don't regress on this

> Hi [@TK-21st](https://github.com/TK-21st), I've opened a PR to fix this at [#36404](https://github.com/huggingface/transformers/pull/36404). After investigation, the error you've found is very template-specific. Can you tell me what template you were using? I'll add a test using it to the `transformers` codebase so we don't regress on this

Great thanks!
I was using `google/gemma-2-2b-it`
tokenizers.apply_chat_template with continue_final_message=True with </think> token
Author: isamu-isozaki
Pull request: no
Labels: name: bug

### System Info

This is very similar to [this](https://github.com/huggingface/transformers/issues/35433) issue but since it's slightly different I made a new issue. This is on transformers 4.49.0
I was working with ```deepseek-ai/DeepSeek-R1-Distill-Qwen-32B``` and I found that the below gives an exception saying substring not found
```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("deepseekr1_qwen_32b")
history = [
           {'role': 'assistant', 'content': "</think>"},
]
continue_final_message = True
tokenizer.apply_chat_template(history, tokenize=False, continue_final_message=continue_final_message)
```
I didn't look too much into the cause but I did notice this happens whenever the model response ends with the </think> token which does happen occasionally(maybe a hallucination)

### Who can help?

@ArthurZucker and @itazap

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("deepseekr1_qwen_32b")
history = [
           {'role': 'assistant', 'content': "</think>"},
]
continue_final_message = True
tokenizer.apply_chat_template(history, tokenize=False, continue_final_message=continue_final_message)
```
this will raise an error. The tokenizer folder was made with a folder named deepseekr1_qwen_32b with the tokenizer json files from https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-32B/tree/main

The output is an exception saying substring not found

### Expected behavior

No exceptions and no output

Comments:
Taking this one! Very likely the Deepseek template is stripping special tokens somewhere

Hi @isamu-isozaki this error is caused by the following line in `tokenizer.chat_template`:

```
{% if '</think>' in content %}{% set content = content.split('</think>')[-1] %}
```

This causes your message content to be removed when the template is applied, and this causes the `continue_final_message` logic to fail (because the message it's trying to continue has been deleted).

We can't really "fix" this, because it's part of the template's behaviour! However, maybe I can add a more informative error message.

@Rocketknight1 Thanks for the quick response! And ah, makes perfect sense. Thanks for investigating. Just checked the pr and I think that error message makes more sense.
Fix edge case for continue_final_message
Author: Rocketknight1
Pull request: yes

The code for `continue_final_message` in `apply_chat_template` fails in the following edge case:

- The chat template trims trailing spaces from messages
- The chat template adds its own spacing after messages
- The unstripped content of the final message is identical to the stripped content of a previous message plus the template's built-in spacing

This causes the template to incorrectly identify the previous message as the final message and strip off a big chunk of the conversation after this point. This is a fairly uncommon edge case, but it's worth fixing nonetheless! cc @TK-21st

This PR handles things more elegantly without a try-except block, and also improves handling when assistant messages contain text/image blocks (which is rare right now, but will probably become more common in future)

Fixes #35433 
Fixes #36440

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36404). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

cc @cyrilvallez @ArthurZucker for core maintainer review!

Also added a clearer error message for the case described in #36440 - this should also be quite rare.

Also, please make sure to rebase to properly trigger tests as this one also has its branch pointing just after the faulty commit!
[Style] fix E721 warnings
Author: kashif
Pull request: yes

# What does this PR do?

Fix E721 Use `is` and `is not` for type comparisons and do not print `config.hidden_size` as a tuple.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36474). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

i am good!


@kashif sorry for the failing tests - we merged a PR that fixes them and I did a rebase!
Remove unused code
Author: Rocketknight1
Pull request: yes

The tokenizer `from_pretrained` code has a dead code path (a list that is never appended to), so this PR removes it!

Comments:
Hi 👋, thank you for opening this pull request! The pull request is converted to draft by default. When it is ready for review, please click the `Ready for review` button (at the bottom of the PR page).

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36459). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
ERROR: Video features and Video Tokens do not match!!!
Author: Diezz01
Pull request: no
Labels: name: bug

name: Multimodal

name: VLM

Hi, i tried to fine tune LLava Next Video using a custom Dataset, but every time i try to train the model i receive this error in output. I followed the guide for the fine tuning here https://github.com/NielsRogge/Transformers-Tutorials/blob/master/LLaVA-NeXT-Video/Fine_tune_LLaVa_NeXT_Video_with_HFTrainer.ipynb. 
I do not understand why i receive this error because if i pass a video composed by only one frame the model accept it and start the training. Actually I am using the Atlas Dione Dataset to fine tune this model.
This is my notebook for the fine tuning:
https://github.com/nicol-buratti/activity-recognition/blob/main/llava_finetuning.ipynb
this is the script for the dataset preparation:
https://github.com/nicol-buratti/activity-recognition/blob/main/activity_dataset.py

Thank you all!

Comments:
@Diezz01 hey! Those train scripts actually would need an update given we've been changing a lot of things since the model was released. But I think they wouldn't fail with token mismatch

Which transformers version you are using and can you provide a small reproducer with an input video (can be even random array)?



I tried with version 4.48.1 and then installed via github link, looks like that increasing the MAX_LENGTH variable it accept some frames of a video. All the videos have the same frame size but they differs from the length

@Diezz01 Ah I see, yes you'd have to increase the `max length` to a large number because the `input_ids` now include palceholders for video embeddings, as much as there will be video tokens. So make sure to add around 500 tokens or more per video 

And yeah, I'll keep this issue open to not forget update the demo finetuning notebooks

Thanks I understand the problem, I set the `max_lenght` to 4096 because I have diffrent length videos. Could it be a problem?? if I pass all videos with the same length (30 frames) the model starts the training otherwise if I pass the entire videos I recieve the same error.

I cannot recall now how many tokens each video frame needs in Llava Next Video, but I believe if you take the whole video as input without sampling frames, then the input is expected to be very large. I would recommend to not set the `max length` if you do not want the video tokens to be truncated, or reduce the number of frames if you wanted to save computation

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Add EVEv2 : an Encoder-free VLM
Author: sbucaille
Pull request: no
Labels: name: New model

### Model description

Hi,

EVEv2 is an encoder-free VLM that achieves great results in its category.
It is based on Qwen2.
If no one is already working on this, I'd like to take this as an opportunity to get more familiar with VLM's

### Open source status

- [x] The model implementation is available
- [x] The model weights are available

### Provide useful links for the implementation

Paper: http://arxiv.org/abs/2502.06788
Code: https://github.com/baaivision/EVE/tree/main
Weights: https://huggingface.co/BAAI/EVE-7B-HD-v2.0

Comments:
@sbucaille can I work on this issue?


Hi @Tanuj-rai , sorry I've already started working on this in my PR
Add `Tensor Parallel` support for ALL models
Author: ArthurZucker
Pull request: no
Labels: name: Feature request

name: Tensor Parallel

name: Good Difficult Issue

Just opening this to add support for all models following #34184

Lets bring support to all model! 🤗 

- [x] Llama

It would be great to add the support for more architectures such as
- [ ] Qwen2
- [ ] QwenVl
- [ ] Mistral
- [ ] Llava


... and many more

For anyone who wants to contribute just open a PR and link it to this issue, and ping me for a review!! 🤗 

Comments:
If it's okay, I want to take the Gemma and Gemma2 @ArthurZucker 

@ArthurZucker I will add support for Granite. Thanks

Hey @ArthurZucker! I am going to work on Mistral.

Hey @ArthurZucker! I am going to start working on Qwen2

For tensor parallel, when changing the dimension while applying the heads, why is num_heads made to -1. This essentially means that the head computations are being distributed, which is fine. But, an alternate approach could be to keep the num_heads same per device, and let head_dim be dynamic (-1). Is there a problem with the latter?


> For tensor parallel, when changing the dimension while applying the heads, why is num_heads made to -1. This essentially means that the head computations are being distributed, which is fine. But, an alternate approach could be to keep the num_heads same per device, and let head_dim be dynamic (-1). Is there a problem with the latter?

I think it might affect the rotatory embeddings so it's better to split the heads 

Hey @ArthurZucker! I will add support for Falcon.
Add aya
Author: ArthurZucker
Pull request: yes
Labels: name: New model

# What does this PR do?
Adds aya model 

Comments:
@ArthurZucker @yonigozlan Thank you so much for all your help! Y'all legends!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36521). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Adds GGUF support for Gemma models
Author: kinshuk1207
Pull request: yes

Extends GGUF compatibility to include Gemma alongside existing Gemma2 support
#33260 

- Adds Gemma configuration mapping
- Reuses Gemma2 tensor processor for Gemma models
- Includes test coverage for Gemma GGUF integration


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.




Comments:
Can you take a look at this PR @SunMarc 
Thanks!

Could you fix the conflicts and do `make style` @kinshuk1207. I'll merge the PR afterwards
Add support for Molmo
Author: fakerybakery
Pull request: no
Labels: name: New model

name: Feature request

name: Vision

name: Multimodal

### Feature request

Hi,
Would it be possible to add support for [Molmo](https://huggingface.co/allenai/Molmo-7B-D-0924) (currently using custom code)?
Thanks!

### Motivation

Molmo is not supported

### Your contribution

N/A

Comments:
Thanks for the request! Given the interest of the molmo model, yes, we'll add it to `transformers` very soon.

@LysandreJik that's amazing! I think it may be easiest to have a separate folder for Molmo and MolmoE, where only the latter includes MoE-specific things like the MoEBlock, load balancing loss etc like in https://github.com/huggingface/transformers/blob/main/src/transformers/models/olmoe/modeling_olmoe.py

Hi @LysandreJik! I would like to contribute into Molmo adding. Would it be possible to participate in this task somehow? 

I'm going to ping @molbap and @ArthurZucker, I believe they wanted to sprint it sometimes soon to ensure the best support available. Thanks for offering your help @VladOS95-cyber!

'MolmoProcessor' object is not callable--how to rectify this error

Hi, I'm resuming #33962 soon FYI, we were on a team offsite and now I'm sick hence the lack of activity you've probably seen :) 
@PadmajaVaishnavi currently the model is not supported on `main`, but you should be able to use the weights and processor as they exist on the hub. 

Just wanted to learn about the current status of Molmo integration: Is it ready-to-use  or WIP?  If the latter, any expectation when it may be ready?   

The model's page   https://huggingface.co/allenai/MolmoE-1B-0924  suggests to call  `processor.process(images=...., )` for Molmo. IIUC, it's non-standard:  normally processor is `Callable`  i.e., `processor(images=...., )` for other models

Hey @xrdaukar , WIP, this week I'm not on any sprint/release/other distractions so will see it through 💪 

Thank you for your work on this @molbap - do you think it might be ready this week?
Groq inference provider
Author: VladOS95-cyber
Pull request: no
Labels: name: Feature request

### Feature request

Since several inference provider APIs were added to HF. It would be beneficial to add Groq inference provider as well to expand a list of supported providers.

### Motivation

Groq is a cutting-edge AI inference provider that has recently gained significant attention for its exceptional performance and speed. The company's Language Processing Unit (LPU) Inference Engine offers several key benefits and advantages:
1. Groq's unique approach to AI inference is based on several innovative features like SRAM onlz architecture, co-located compute and memory, kernel-less compiler.
2. Over 500,000 developers are currently using Groq's API keys.
3. It achieved up to 18x faster output tokens throughput compared to other cloud-based inference providers in the Anyscale LLMPerf Leaderboard.
4. Models support: Llama 3 series (8B, 70B, vision models), Mixtral 8x7B, Gemma 2 9B, Whisper Large V3 and so on.

### Your contribution

I could try to dive deep into this topic and provide implementation myself.

Comments:
@ArthurZucker What do you think?

Hi @VladOS95-cyber, unfortunately inference providers have to be added via an agreement between HF and the provider, so users can't manually add Groq. However, if you'd like to use Groq as an inference provider, please let them know! We're fans of their high throughput as well, so it'd be great to have them in the UI!

@Rocketknight1 Hi! Ok, got it, thank you for the answer!