noamaanMulla-03/transformers-issues huggingface dataset:
[Feature request] FastV: Plug-and-play inference acceleration  for  large vision language models
Author: chenllliang
Pull request: no
Labels: name: Feature request

name: Vision

name: optimization

### Feature request

FastV is a plug-and-play inference acceleration method for large vision language models relying on visual tokens. It could reach 45% theoretical FLOPs reduction without harming the performance through pruning redundant visual tokens in deep layers. It is originally implemented in https://github.com/pkunlp-icler/FastV  , with [paper](https://arxiv.org/abs/2403.06764) and [demo](https://www.fastv.work/)

![fastv_tradeoff](https://github.com/huggingface/transformers/assets/43977888/97490036-8942-440f-b46b-5629e68d817a)

*aokvqa results*
| Model                         | Score | latency / first output token (A100 80G) | GPU Memory |
| ----------------------------- | ----- | --------------------------------------- | ---------- |
| 7B Vanilla Decoding           | 76.8  | 0.138s                                  | 18G        |
| 13B Vanilla Decoding          | 81.9  | 0.203s                                  | 33G        |
| \- 13B FastV (K=2 R=25%)      | 81.8  | 0.181s                                  | 29G        |
| \- 13B FastV (K=2 R=50%)      | 81.3  | 0.155s                                  | 28G        |
| \- 13B FastV (K=2 R=75%)      | 80.9  | **0.124s**                                  | 27G        |
| 13B Vanilla Decoding 4Bit     | 81.5  | 0.308s                                  | 12G        |
| \- 13B FastV 4Bit (K=2 R=25%) | 81.7  | 0.277s                                  | 11G        |
| \- 13B FastV 4Bit (K=2 R=50%) | 81.1  | 0.275s                                  | 10G        |
| \- 13B FastV 4Bit (K=2 R=75%) | 80.3  | 0.245s                                  | **9G**         |

### Motivation

I want to merge the FastV feature to speed up the inference of HF's multimodal models (Llava is the first one).  Since Llava is based on Llama, I need to add features to both the Llama and Llava model. 

I would add a fastv_config parameters to the modeling_llava.py from llava and add a new forward function named fastv_forward from llama. I want to know if it is ok to do that or I should create a new model.

### Your contribution

I would submit a PR. 

Comments:
UmT5 Flax modelling
Author: long21wt
Pull request: no
Labels: name: New model

name: Good Second Issue

name: Flax

### Feature request

I'd like to have the UmT5 Flax modelling file, there was a PR for it https://github.com/huggingface/transformers/pull/22626 but was closed by https://github.com/huggingface/transformers/pull/24477 PR. The new one only has UmT5 PyTorch modelling file.

### Motivation

Since UmT5 is originally implemented in Jax and Flax, it would make sense to have the flax modelling file for the model.

### Your contribution

I could help testing the modelling file

Comments:
cc @sanchit-gandhi 

I sadly won't have time to contribute the Flax UmT5 model myself, but would be more than happy to assist anyone who wants to give the integration a go themselves! We could largely leverage the Flax T5 modelling code, so it could be quite a fast addition: https://github.com/huggingface/transformers/blob/main/src/transformers/models/t5/modeling_flax_t5.py

So let's open this one up to the community and call on any Flax contributors who would like to have a go at a quick encoder-decoder model addition!

For sure! As mentioned above, I would copy the entire Flax T5 modelling script, then try to make the minimum changes required to update it to UmT5. This is because the Flax T5 modelling code is already heavily optimised and in the Transformers format, so will speed up the integration significantly. 

Feel free to open a PR and ping me for a review! Happy to answer any questions/queries :)

Hi @sanchit-gandhi, I have started working on this. Should I create a WIP PR?
[`causal_mask`] Add a warning when registering the buffer
Author: ArthurZucker
Pull request: yes

# What does this PR do?

After #27931, the importance of `max_position_embedding` has been greatly improved. Before it would cache `sin_cache` and `cos_cache`, which are relatively small tensors. 
With the update and the need of a static shaped mask, we are no longer caching sin and cos, but are caching the `causal_mask`. 

1. Either we keep this and warn users
2. Or we allocate a new mask at each forward, making sure this is compatible with compile. 

Decided to go for 1.

One possible problem is if we re-introduce the `sin_cache` and `cos_cache` for performance issues (if it proves to be a lot slower). 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29742). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Tests] Remove unused code
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR removes some code that has been commented out for 5 years. Seen in #29586. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29737). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Incorrect jinja2 package version check
Author: Ricardo-L-C
Pull request: no

### System Info

- `transformers` version: 4.37.0
- Platform: Linux-4.14.105-1-tlinux3-0013-x86_64-with-glibc2.10
- Python version: 3.8.8
- Huggingface_hub version: 0.20.3
- Safetensors version: 0.4.2
- Accelerate version: 0.23.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. installing package `jinja2==3.0.0`
2.
```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen1.5-72B-Chat")

prompt = "Give me a short introduction to large language model."
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
```

will raise an error:
```
.../transformers/tokenization_utils_base.py in _compile_jinja_template(self, chat_template)
   1769 
   1770         if version.parse(jinja2.__version__) <= version.parse("3.0.0"):
-> 1771             raise ImportError(
   1772                 "apply_chat_template requires jinja2>=3.0.0 to be installed. Your version is " f"{jinja2.__version__}."
   1773             )

ImportError: apply_chat_template requires jinja2>=3.0.0 to be installed. Your version is 3.0.0.
```

There is a logical conflict in the error message, due to line 1770 shows `version <= 3.0.0` will raise this ImportError.

### Expected behavior

change the `if` condition to:
```python
if version.parse(jinja2.__version__) < version.parse("3.0.0"):
```

Comments:
Indeed! Would you like to open a PR to add this? This way you get the github contribution for the solution.

cc @Rocketknight1 

> Indeed! Would you like to open a PR to add this? This way you get the github contribution for the solution.
> 
> cc @Rocketknight1

Thank you for this suggestion. I would like to open a PR asap.

Closing this issue as fixed now that #29754 is merged!
fix jinja2 package version check
Author: Ricardo-L-C
Pull request: yes

# What does this PR do?

Fix jinja2 package version check

Fixes # (issue)
[Incorrect jinja2 package version check #29752](https://github.com/huggingface/transformers/issues/29752)

## Before submitting

- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x]  Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request), Pull Request section?
- [x]  Was this discussed/approved via a Github [issue](https://github.com/huggingface/transformers/issues/29752) or the forum? Please add a link to it if that's the case.
- [ ]  Did you make sure to update the documentation with your changes? Here are the [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x]  Did you write any new necessary tests?

## Who can review?

@amyeroberts 

## Tests

1. install specific jinja2 package with `pip install jinja2==3.0.0`. Other environments have been pointed out in the issue.

2. I test with follow code:
```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen1.5-72B-Chat")

prompt = "Give me a short introduction to large language model."
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": prompt},
    {"role": "assistant", "content":"hello"}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
print(text)
```

And got the desired output instead of raising an error:
```

system
You are a helpful assistant.


user
Give me a short introduction to large language model.


assistant
hello

```

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29754). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Support sharded safetensors in TF
Author: Rocketknight1
Pull request: yes

Right now our TF safetensors loading doesn't support sharded checkpoints, which is a problem as more and more big models move to safetensors weights only! This is currently blocking @a8nova's PR at #26870.

As sharded safetensors **saving** for TF was also missing, this PR adds that as well, and expands the tests to cover both.

TODO:

- [x] Check we prioritize safetensors vs. TF weights correctly when both are present (and make sure behaviour matches for sharded vs. unsharded)
- [x] Add tests for sharded safetensors loading from both PT and TF format
- [x] Add test for saving sharded safetensors

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29350). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This should be ready for review! cc @ArthurZucker @a8nova

@LysandreJik I think now that we have proper support we can switch to safetensors by default immediately, either in this PR or in a follow-up. 

I'd wait for a few weeks just to ensure we don't have reports of failure and switch for the next version. WDYT?

Sounds good to me!

@LysandreJik is there anything else to be resolved before I merge this? (Except for the failing test, but that's not specific to this PR)

All comments addressed @amyeroberts! I think we should be ready now.

This looks ready to go, cc @a8nova! We have a branch cut + release planned on Monday, though, and since this touches a lot of core code I don't want to merge it right before a release. Instead, I suggest merging it right after the branch cut, and then we can finalize and merge the PRs that are blocked by it: TF-IDEFICS, TF-Gemma and possibly the Mistral/Mixtral PRs if @ariG23498 can have one of them ready by then (no stress, obviously!)

Then we could launch all the new Keras models together in the following release and do a section in the release notes about them, crediting @a8nova and @ariG23498?

cc @a8nova and @ariG23498, this has now been merged. If you rebase your PRs, that should resolve any issues with sharded safetensors loading!
Update Security Policy to provide Security Guidance for users
Author: diogoteles08
Pull request: yes

# What does this PR do?

Fixes #29256

As suggested on the above issue, this PR complements your current Security Policy by also providing some guidance on how to use Transformers in the most secure way.

Additionally, I acknowledge that HuggingFace have some extensive documentation, so if you'd rather keep those security guidance elsewhere, let me know!

Thanks!


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

As suggested on the PR template, I'm tagging the two members responsible for documentation, as I believe would be a good fit for this change

@stevhliu and @MKhalusova

Comments:
Hello @diogoteles08, thank you for your PR.

I am currently revamping the security policy for `transformers` which, indeed, needs to be updated. I'll link your PR from there, and if I end up re-usign some of your work, I'll make sure to add you as co-author.

Thank you!

I went for something simpler here: https://github.com/huggingface/transformers/pull/29707

Would you like me to add you as co-author to the PR?

> Would you like me to add you as co-author to the PR?

Well, if my PR has provided any help to build yours, I'd be honored to be added as a co-author =)

And thanks for updating the status here, btw. I'll close this PR and follow up on your PR, if I have anything to add.
Add correct batched handling for apply_chat_template
Author: Rocketknight1
Pull request: yes

`apply_chat_template` has a few issues since it was written. Firstly, by default it returns the naked `input_ids` rather than a dict, and secondly it didn't support rendering a batch of chats simultaneously. This PR makes a few changes:

- Batched chats are now supported, and we sniff the input to figure out what the user is passing
- `return_dict` now defaults to `None`. For now, we interpret this as `False` to maintain backward compatibility, but this PR adds a warning that the default behaviour will be changing to `True` to match other tokenizer methods.

cc @siddk @lewtun who have both requested this or something like it!

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29222). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Should be ready for review now! cc @ArthurZucker 

This should be ready for re-review now, cc @amyeroberts @ArthurZucker! I simplified the PR by removing the deprecation warning - I'm not sure if we want to move to `return_dict=True` that quickly anyway. As a result, this shouldn't result in any behaviour changes now, it only adds new functionality.

Merging this now that the branch cut has passed!
Fix docker image build
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix docker image build.

See https://github.com/huggingface/transformers/actions/runs/8363449527

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29762). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix RoPE config validation for FalconConfig + various config typos
Author: tomaarsen
Pull request: yes

# What does this PR do?

One change that fixes a bug:
* Fix RoPE config validation: `config = FalconConfig(alibi=False, rope_scaling={"type": "linear", "factor": 2.0})` would fail. (1fbd6993b8ac0c5dee1b26656648ee8360c06aa3)

And three changes that fix typos/improve maintainability etc.
* Fix code block tag (b9e4f0a826792ba7bd4ed20d810a0311ac76890f)
* Fix typos: an float -> a float (69214ce56aeeb450c6deb49c3365048b99225436)
* Implement copy functionality for Fuyu and Persimmon (4ee278a6a5c904feee02df7eaaddb91d11839d00)

I can split this up into two smaller PRs if preferred - just let me know.

## Details
### Fix RoPE config validation
```python
from transformers import FalconConfig

config = FalconConfig(alibi=False, rope_scaling={"type": "linear", "factor": 2.0})
```
Running this on main produces:
```
ValueError: `rope_scaling` is not supported when `alibi` is `True`.
```
Which is obviously nonsensical. This is caused by `if self.rotary: raise ValueError` where `self.rotary` is just `not self.alibi`. So, this ValueError is thrown when alibi is `False`, not when it is `True`.

The other 3 changes speak for themselves.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

No new tests - I can add one for the first bug fix if preferred.

## Who can review?

@gante 

- Tom Aarsen

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26929). All of your documentation changes will be reflected on that endpoint.
[docstring] Fix docstring for ChineseCLIPImageProcessor, ChineseCLIPTextConfig, ChineseCLIPVisionConfig
Author: Sparty
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes https://github.com/huggingface/transformers/issues/26638


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->

Comments:
@ydshieh Could you please review this?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26880). All of your documentation changes will be reflected on that endpoint.
Converting CLIPModel to VisionTextDualEncoderModel
Author: fabiozappo
Pull request: no

Hi HF team and thanks for your amazing work, for my research I would like to use the CLIPModel "openai/clip-vit-base-patch32" through the VisionTextDualEncoderModel class. I've tried to use the `from_pretrained()` method but it doesn't support the `clip_text_model` class  of the text tower.

Do you have any suggestion from where to start to create such a script?

Comments:
Hi @fabiozappo, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

Ups, thanks!

I've created a question [here](https://discuss.huggingface.co/t/converting-clipmodel-to-visiontextdualencodermodel/78420)

Closing :)
[docs] Remove redundant `-`  and `the` from custom_tools.md
Author: windsonsea
Pull request: yes

Minor changes about redundancy removal.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29767). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
OWL-ViT box_predictor is wildly inefficient since box bias is not precomputed 
Author: 5hadytru
Pull request: no
Labels: name: Good First Issue

name: Vision

### System Info

The box_predictor function computes box bias from scratch each time it is called, but box bias does not actually depend on anything besides the shape of the feature map (which depends on the batch size (which can be broadcasted), number of ViT patches, and ViT token dim) and therefore should be precomputed. This is a super simple fix and will result in a >>10x inference speedup. (https://github.com/huggingface/transformers/blob/ce2e7ef3d96afaf592faf3337b7dd997c7ad4928/src/transformers/models/owlvit/modeling_owlvit.py#L1389C35-L1389C35)

### Who can help?

@amyeroberts @pasqualedem @alaradirik 

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Steps to reproduce the behavior:

1. Add some lines of code to box_predictor which time its overall execution + the execution of each individual line
2. Run one of the inference examples

### Expected behavior

compute_box_bias will take a substantial amount of time (due to the normalize_grid_corner_coordinates call) while the other lines will be negligible

Comments:
Hi @5hadytru, thanks for raising this issue! 

Indeed - this isn't ideal. Would you like to open a PR to make the bias calculation a bit more sane and efficient? 

cc @rafaelpadilla 

Hi. I would love to help if no one else is working on this.

@nvbinh15 Great! It's all yours :)