noamaanMulla-03/transformers-issues huggingface dataset:
PeftModel is not an instance of PreTrainedModel.  `No liger kernels will be applied.`
Author: ambroser53
Pull request: no
Labels: name: bug

name: optimization

name: PEFT

### System Info

`transformers==4.45.1`
`peft==0.13.0`
`liger-kernel==0.3.1`


So `isinstance(model.base_model.model, PreTrainedModel)` returns true but `isinstance(model, PreTrainedModel)` returns false so no liger kernels are used.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Load any model with a PeftModel wrapper via `get_peft_model` and try to run with the `use_liger_kernel` flag in the trainer.

### Expected behavior

Apply liger kernels. Be as simple as add a check for peft models that then checks `model.base_model.model`

Comments:
cc @benjaminbossan for the `PeftModel` and @ArthurZucker for the liger kernel

I've narrowed the cause down to the changes made in #33502

Just to confirm, a `PeftModel` is not an instance of `PretrainedModel`, so it is expected that this check would return `False`. As @ambroser53 correctly identified, we can [check if the model is a PEFT model](https://github.com/huggingface/transformers/blob/0dbc7090bac3fb50f679a8a93516bdc67019c997/src/transformers/trainer.py#L255) and if this is true, we can check:

1. `isinstance(model.base_model, PreTrainedModel)` -> apply liger to `model.base_model` (for prompt learning methods)
2. `isinstance(model.base_model.model, PreTrainedModel)` -> apply liger to `model.base_model.model` (for LoRA etc.)

I don't know what exactly liger kernels do, but from the description

> We have implemented Hugging Face Compatible RMSNorm, RoPE, SwiGLU, CrossEntropy, FusedLinearCrossEntropy

they should not interfere with PEFT and should be usable in conjunction. LMK what you think.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Why has this been closed? It's not fixed and it's a very easy change. I've been having to manually do it this whole time on my local transformers version.

@ambroser53 The issue has been closed due to in inactivity, but as you mentioned, it's still not resolved.

> I've been having to manually do it this whole time on my local transformers version.

Would you be interested in providing your fix in a PR?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix: Falcon tie_word_embeddings in GGUF
Author: MekkCyber
Pull request: yes

# What does this PR do?
In the `modeling_gguf_pytorch_utils.py` file, the value of `tie_word_embeddings` is determined by the presence (or absence) of the tensor `output.weight` in the GGUF file. While this approach is generally a good indicator, the Falcon architecture is an exception to the rule as you can see here : https://huggingface.co/tiiuae/falcon-7b/tree/main?show_file_info=model-00002-of-00002.safetensors (hf format) and here https://huggingface.co/tensorblock/falcon-7b-GGUF/tree/main?show_file_info=falcon-7b-Q2_K.gguf (gguf format)

In Falcon, `word_embeddings` are tied to the `lm_head` weights. Despite this, `output.weight` is still present in the GGUF file, and `lm_head` is included in the Hugging Face model format. To handle this edge case, I added an exception array for such architectures.

This issue was causing a subtle error related to parameters not being on the same device, which was only discoverable in multi-GPU settings.

## Who can review ?
@SunMarc @Isotr0py 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35715). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

But I remember `tie_word_embeddings` is not used by `modeling_falcon.py`? 

BTW, if the issue about `tie_word_embedding` is the case, can we also update the falcon conversion test to cover this edge case?

`tie_word_embeddings` is not used directly in the modeling file of the model, instead it's used in `modeling_utils.py` : https://github.com/huggingface/transformers/blob/main/src/transformers/modeling_utils.py#L1858
It was the failure of the Falcon conversion test that made me realize the error, and now it passes successfully.
Break point in Transformers V4.48.0 and python 3.9
Author: prajeet26
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.48.0
- Platform: Linux-5.15.0-25-generic-x86_64-with-glibc2.35
- Python version: 3.9.0
- Huggingface_hub version: 0.27.1
- Safetensors version: 0.5.2
- Accelerate version: 1.2.1


### Who can help?

 @gante  @Rocketknight1

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction


Install Python 3.9
Install Transformers >= v4.48.0
install fastchat accelerate 
I was trying to attach delta weights of Vicuna-13B to existing Llama 13B weights in local , for this the advised was below:
```
python3 -m fastchat.model.apply_delta \
    --base-model-path /path/to/llama-13b \
    --target-model-path /path/to/output/vicuna-13b \
    --delta-path lmsys/vicuna-13b-delta-v1.1
```
doing this raised the following error:
```
RuntimeError: Failed to import transformers.generation.streamers because of the following error (look up to see its traceback):
unsupported operand type(s) for |: 'type' and 'NoneType'
```


### Expected behavior

Work in Python 3.9

Comments:
[watch this](https://github.com/huggingface/transformers/pull/35103)

#35725 fixes it :)
Improve image processing time
Author: yonigozlan
Pull request: no
Labels: name: Feature request

name: Vision

name: optimization

name: Processing

### Feature request

Optimize Transformers' image_processors to decrease image processing time, and reduce inference latency for vision models and vlms.

### Motivation

The Transformers library relies on PIL (Pillow) for image preprocessing, which can become a major bottleneck during inference, especially with compiled models where the preprocessing time can dominate the overall inference time.

![image](https://github.com/user-attachments/assets/10e19b7a-5e48-4bef-84a1-430458e17586)
![image-1](https://github.com/user-attachments/assets/be4f06f3-befd-44b7-b743-10f09471fa58)


In the examples above, the RT-DETR preprocessing necessitates only to resize the image, while the DETR one involves resize+normalize. 
In eager mode, image preprocessing takes a big part of the total inference time for RT-DETR, but is not the main bottleneck. However, with a compiled RT-DETR, image preprocessing takes up the majority of the inference time, underlining the necessity to optimize it. This is even clearer for DETR, where image preprocessing is already the main bottleneck in eager mode.

However, alternative libraries exist that leverage available hardware more efficiently for faster image preprocessing.
[OptimVision](https://github.com/yonigozlan/OptimVision) uses such libraries to get much better results compared to Transformers.

Much more details on OptimVision and image processing methods comparison are available on this [Notion page](https://www.notion.so/huggingface2/OptimVision-Optimize-preprocessing-time-10f1384ebcac8091a12debb87fe5f591?pvs=4).

### Your contribution

OptimVision is an experiment playground to optimize the different steps involved in inferring/training with vision models.
The current fast image preprocessing in OptimVision is a proof of concept and is not yet ready to be merged into Transformers, but that this the ultimate goal :).

Comments:
Sounds like a good project indeed :)

Hi, any updates on making Qwen2VLProcessorFast, as we are experiencing a major bottleneck because of the preprocessing time in offline mode

Answered here https://github.com/huggingface/transformers/issues/34272#issuecomment-2596008142
Fix the bug that `Trainer` cannot correctly call `torch_jit_model_eval`
Author: Wanguy
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fix the bug that the accelerator.autocast does not pass parameters correctly when calling torch_jit_model_eval (#35706)

> `Accelerator.autocast` no longer accepts a cache_enabled arg, users must make an `AutocastKwargs` instance and pass it in as a kwarg handler. (from [🚨🚨🚨 The Great Deprecation 🚨🚨🚨](https://github.com/huggingface/accelerate/pull/3098))


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
https://github.com/huggingface/transformers/issues/35706
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@SunMarc



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35722). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add proper jinja2 error
Author: Rocketknight1
Pull request: yes

Several users reported confusion in #34397 when `jinja2` is not installed, because the code imports some submodules of `jinja2`, and the error message when these submodules is undefined doesn't reference `jinja2` at all.

This PR adds a proper error so people know what's going on!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35533). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

failing test is unrelated, pinging @arthurzucker for core maintainer review!

It's a dependency of `torch` so it gets installed almost all of the time anyway, but maybe we should just give up and make it a full dependency of `transformers` at some point
jinja2 is a necessary dependency, but it is not currently specified
Author: FlyingCarlos
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.45.2
- Platform: Linux-3.10.0-1160.114.2.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.10.15
- Huggingface_hub version: 0.26.1
- Safetensors version: 0.4.5
- Accelerate version: 1.0.1
- Accelerate config:    not found
- PyTorch version (GPU?): 1.12.1+cu102 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I have addressed the issue: if Jinja2 is not installed, an ‘Extension’ undefined error will occur.

### Expected behavior

NA

Comments:
https://github.com/huggingface/transformers/blob/30c76d5b2836b9ba6b0e417aafb5ba77b4129ffe/src/transformers/utils/chat_template_utils.py#L362-L366



And "Extension" is from jinja2

https://github.com/huggingface/transformers/blob/30c76d5b2836b9ba6b0e417aafb5ba77b4129ffe/src/transformers/utils/chat_template_utils.py#L28-L33

This is a design decision to keep the number of installed packages small. It's listed here: https://github.com/huggingface/transformers/blob/main/setup.py#L127. As you see, not even torch is installed by default: https://github.com/huggingface/transformers/blob/main/setup.py#L184, https://github.com/huggingface/transformers/blob/main/setup.py#L423

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Thanks for flagging this. I was getting the 'Extension' undefined error and was quire confused until I found this.

Likewise, I ran into the same problem - really would be wonderful if there was at least a specific error message that directed the user to add jinja2 as a dep!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

There should be a proper error message now (#35533)!
Modular: support for importing functions from any file
Author: Cyrilvallez
Pull request: yes

# What does this PR do?

Allow to import function definition from file unrelated from the inheritance tree. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35692). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Remove batch size argument warning when unjustified
Author: quintenroets
Pull request: yes

# What does this PR do?

Fixes [#35518](https://github.com/huggingface/transformers/issues/35518)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@ArthurZucker 
@zucchini-nlp 

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Batch size deprecation warning issued even when it is not used
Author: quintenroets
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.48.0.dev0
- Platform: Linux-6.11.0-13-generic-x86_64-with-glibc2.40
- Python version: 3.12.7
- Huggingface_hub version: 0.27.0
- Safetensors version: 0.5.0
- Accelerate version: 1.2.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.4.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed


### Who can help?

@ArthurZucker @zucchini-nlp 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoModel
from transformers.cache_utils import StaticCache

model_id = "deepset/roberta-base-squad2"
model = AutoModel.from_pretrained(model_id)
cache = StaticCache(max_batch_size=10, config=model.config)

```

**Output**
`The 'batch_size' attribute of StaticCache is deprecated and will be removed in v4.49. Use the more precisely named 'self.max_batch_size' attribute instead.
`

### Expected behavior

No warning message 

Comments:
[cache] add a test to confirm we can use cache at train time
Author: gante
Pull request: yes

# What does this PR do?

In #35648, the idea of always disabling the cache at train-time was floated. However, @BenjaminBossan pointed to me that some fine-tuning scripts need cache at train time (e.g. prefix tuning in PEFT).

I couldn't find any test that ensured the survival of this property, so this PR adds a test that will fail if we decide to disable it :) 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35709). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks a lot for adding this test to prevent possible regressions with PEFT methods like prefix tuning. I wonder if we can also try injecting the cache and checking if that works, basically like a mini prefix tuning:

```python
        # simulate injecting virtual tokens like in prefix tuning
        num_virtual_tokens = 3
        past_key_values = [torch.randn(2, 1, 2, num_virtual_tokens, 8)] * 2
        past_key_values = DynamicCache.from_legacy_cache(past_key_values)
        model_inputs["attention_mask"] = torch.cat((
            model_inputs["attention_mask"],
            torch.ones(1, num_virtual_tokens).to(model_inputs["attention_mask"].device)
        ), dim=1)
        model_outputs = model(**model_inputs, past_key_values=past_key_values, use_cache=True)
```

Maybe this is overkill, LMK what you think.

@BenjaminBossan added, thank you for the suggestion 👍 (the extra check is nearly free, and may save some pain down the road)
🧹 remove `generate`-related objects and methods scheduled for removal in v4.48
Author: gante
Pull request: yes

# What does this PR do?

See title 🧹 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35677). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Remove `pt_to_tf`
Author: gante
Pull request: yes

# What does this PR do?

See title -- this command was scheduled to be removed in v4.47

(part of release-related 🧹, deleting things I work on that have been tagged for removal) 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35672). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[generate] can instantiate `GenerationConfig(cache_implementation="static")`
Author: gante
Pull request: yes

# What does this PR do?

See title -- we should be able to instantiate `GenerationConfig(cache_implementation="static")`, to configure `generate` calls with static caches. In other words, we shouldn't need any further parameterization, as it can be inferred at inference time.

Fixes #35026 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35679). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[generate] return Cache object even if passed in a legacy format
Author: gante
Pull request: yes

# What does this PR do?

See title. Legacy behavior scheduled to be removed in v4.47.

We now prioritize returning the same cache format as the model (usually a `Cache` object), instead of converting to the legacy format. This used to happen if the user passed in a legacy cache.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35673). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.