noamaanMulla-03/transformers-issues huggingface dataset:
Llama et al. / FSDP : Fix breaking change in 4.40 for FSDP
Author: amyeroberts
Pull request: yes

See #31161 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31638). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Closing as I was able to update the original PR through the online editor in the end
Add LLaVa NeXT Video
Author: zucchini-nlp
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Adds a new model for Video LLMs built by tuning LLaVa-NeXT on video dataset, current SOTA for videos on [VideoMME bench](https://arxiv.org/abs/2405.21075).

This PR introduces some changes to how we usually handled interleaved multimodal data. I need @amyeroberts opinion on that, of it's a viable option for these kinds of models. I added a separate videoProcessor class which has all same parameters as ImageProcessor but a different logic. The reason: imageProcessor file was already full of extra helper functions, and adding more conditions seemed to degrade readbility

Also, I added a chat template, if you can verify that it's correct @NielsRogge 😄 (it's in the `convert_model.py`). I will push all templates to the hub later, currently only 7B model has its template working

Fixes #31164

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31252). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

ready for review!

@amyeroberts ready for re-review

- Diff was not working and wasn't copying content from parent class (LLava_NeXT). I did some changes in the diff-converter file, so would be nice to get @ArthurZucker review on that. However, diff still doesn't work when we have to overwrite docstring for config, so I simply made my own new config file. I also tried diff-converting example2, still doesn't copy docstring and I couldn't solve it yet  
- VideoProcessor is gone, so now we just have two ImageProcessor classes used for each modality, simply to isolate unrelated code in two files

@amyeroberts this one is also ready for the last review

@zucchini-nlp Only thing before merge is to do a run on the slow tests for the model with a `[run_slow] llava-next-video` commit message 

Hmm I am not sure slow tests are working, they were skipped in here and in BlipVideo. I think the commit message format is correct

> Hmm I am not sure slow tests are working, they were skipped in here and in BlipVideo. I think the commit message format is correct

Ah, we didn't have the `run-slow` label on the PR, which might have caused it to be skipped. @zucchini-nlp Could you try again?
LLaVA-NeXT-Video support
Author: AmazDeng
Pull request: no
Labels: name: New model

### Model description

The model to consider.
The llava-next-video project has already been released, and the test results are quite good. Are there any plans to support this project?

Currently, Hugging Face does not support this model.

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

https://github.com/LLaVA-VL/LLaVA-NeXT/blob/inference/docs/LLaVA-NeXT-Video.md

Comments:
Hey! LLava-NeXT-Video is planned to be added to transformers. See more here https://github.com/LLaVA-VL/LLaVA-NeXT/issues/9 :)



> Hey! LLava-NeXT-Video is planned to be added to transformers. See more here [LLaVA-VL/LLaVA-NeXT#9](https://github.com/LLaVA-VL/LLaVA-NeXT/issues/9) :)

So is there an approximate point in time?

I would say approximately mid of June. I will ping the authors to ask about the state of their next release as we've been getting a lot of requests to add this model. I'll give this model higher priority   

> I would say approximately mid of June. I will ping the authors to ask about the state of their next release as we've been getting a lot of requests to add this model. I'll give this model higher priority

ok, thanks.
Generate: fix assisted generation with `past_key_values` passed as kwargs
Author: gante
Pull request: yes

# What does this PR do?

This PR:
- Fixes assisted generation when `generate` is called with the `past_key_values` kwarg (contrarily to other kwargs, this one shouldn't be passed to the assistant model, as it is the cache of the main model)
- Renames `maximum_length` to `max_length` in the newly added `DynamicCache.crop` function (`max_length` is the common variable name to depict a maximum length, standardizes API before the function is released)

Comments:
Thanks for the quick review 💛 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31644). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Generation: past kv can be None
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes #31017. The linked code would work in the prev version, so we have to check if "past_key_values" is not None when obtaining cache position.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31051). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
model.generate() able to accept past_key_values=None
Author: JackCai1206
Pull request: no
Labels: name: Generation

### Feature request

In `model.generate()`, if we pass in `past_key_values=None`, it should behave as if the keyword argument is not there. 

### Motivation

Currently, if we pass in `past_key_values=None`, we would get an error in utils.py. For my use case, I am evaluating a model with prompts of varying context length, but the prefix is the same, so the code looks like this
```
   prompts = ['a', 'ab', 'abc', 'abcd', ...]
   kv_cache = None
    for i in range(len(prompts)):
        prompt = prompts[i]
        inputs = tokenizer(prompt, return_tensors="pt", padding='longest').to(config.device)
        with torch.no_grad():
            if i == 0:
                outputs = model.generate(**inputs, generation_config=generation_config)
            else:
                outputs = model.generate(**inputs, generation_config=generation_config, past_key_values=kv_cache)
```
I want to take advantage of the past_key_values, but in the first iteration of the loop, there is no `past_key_values` yet, so `kv_cache` is `None` and I have to use an if statement to check it. I wanted to avoid the if statement around `model.generate()` by directly passing `past_kv_values=kv_cache` even when the variable is None. 

### Your contribution

in `tranformers/generation/utils.py` L1317, check if `past_key_values is None`. 

Comments:
cc @gante @zucchini-nlp 

@JackCai1206 hey! Thanks for noticing, you're right. This was caused by the latest release and the new cache format. I will open a PR on Monday to fix it
(Have PR) Speed up `BeamScorer` to make GPT-2 generation 2-3x faster
Author: fzyzcjy
Pull request: no

### Feature request

Please refer to https://github.com/huggingface/transformers/pull/21234#issuecomment-2090310005 for details (I create a new issue because I guess the reply to ancient issue will not be seen)

### Motivation

-

### Your contribution

-

Comments:
cc @gante 

Hi @fzyzcjy 👋 

The reason for not merging your suggested PR is still valid, we are currently refactoring `generate` and `beam_search` will look very different from what it is at the moment when we're done :)

I see. May I know when will the refactor be done? (seems many months have passed...) and looking forward to the new code!

@fzyzcjy You can see the tracker [here](https://github.com/huggingface/transformers/issues/28981) 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Meet problems when I use the file src/transformers/models/llama/convert_llama_weights_to_hf.py to transfer LlaMa-7B
Author: wwxxyy1996
Pull request: no

### System Info

![image](https://github.com/huggingface/transformers/assets/161870772/df755d6e-4ac8-4951-9e08-45e625f98952)
When I use the file convert_llama_weights_to_hf.py to converted LlaMa-7B weights to the Hugging Face Transformers format, an error raised for the line 154 and said "RuntimeError: shape '[32, 2, 2, 4096]' is invalid for input of size 16777216".

My command
python convert_llama_weights_to_hf.py     --input_dir ./ --model_size 7B --output_dir ./7B_hf

Here is my environments,
cuda-11.7, gcc-10.2.0, torch==2.0.0+cu117 torchvision==0.15.1+cu117, openai==0.27.8, transformers==4.41.0.dev0

Thank you very much!!

### Who can help?

@ArthurZucker @pcuenca @xe

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Just use this code to convert the model. The model size (consolidated.00.pth file) is 13G.

python convert_llama_weights_to_hf.py     --input_dir ./ --model_size 7B --output_dir ./7B_hf

### Expected behavior

Could you give me some suggestions?

Comments:
Hey! This seems to be a duplicate of #30723, I'll check it out asap 😉 

Thank you very much for your answering!! It is very important for me. Have a good weekend~

Best wishes,
Xinyi

From: Arthur ***@***.***>
Sent: Friday, May 10, 2024 7:10 AM
To: huggingface/transformers ***@***.***>
Cc: Xinyi Wang ***@***.***>; Author ***@***.***>
Subject: Re: [huggingface/transformers] Meet problems when I use the file src/transformers/models/llama/convert_llama_weights_to_hf.py to transfer LlaMa-7B (Issue #30734)

你通常不会收到来自 ***@***.******@***.***> 的电子邮件。了解这一点为什么很重要<https://aka.ms/LearnAboutSenderIdentification>

Hey! This seems to be a duplicate of #30723<https://github.com/huggingface/transformers/issues/30723>, I'll check it out asap 😉

—
Reply to this email directly, view it on GitHub<https://github.com/huggingface/transformers/issues/30734#issuecomment-2103933744>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BGS7HNB3C2DHEWXR42PHNELZBRQCHAVCNFSM6AAAAABHPOETQCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBTHEZTGNZUGQ>.
You are receiving this because you authored the thread.Message ID: ***@***.******@***.***>>

This message and any attachment are intended solely for the addressee and may contain confidential information. If you have received this message in error, please contact the sender and delete the email and attachment. Any views or opinions expressed by the author of this email do not necessarily reflect the views of the University of Nottingham. Email communications with the University of Nottingham may be monitored where permitted by law.


The problem is not solved by changing the transformer version

Why am I marked as a person to help?

No Idea @Xe ! 

And yes I know, I will look into the fix this week! Sorry all 🤗
[`Llama`] Conversion: fix and simplify the script!
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes #30734 #30723 by updating the script. It simplifies the way we call the conversion script, and give better names to variables! Also supports some issues we have had with consolidated names! 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31591). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Skip tests properly
Author: amyeroberts
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Makes sure all tests which are skipped are done so with an explicit skip. This helps with test stats, and doesn't make us think some tests are passing when they're actually empty which can be confusing c.f. [this comment](https://github.com/huggingface/transformers/pull/31495#discussion_r1653085411)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31308). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[QoL] Allow dtype str for torch_dtype arg of from_pretrained
Author: aliencaocao
Pull request: yes

# What does this PR do?

This is to align with the behavior of `pipeline` where users can pass strings like `torch_dtype="float32"` and it gets recognized as `torch.float32`. It now also happens for any `PretrainedModel.from_pretrained`.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@amyeroberts 


Comments:
@amyeroberts added them

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31590). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Missing model license information
Author: guillaume-be
Pull request: no

Hello,

A significant number of models uploaded to the model hub do not contain any license information. I wanted to check if your conditions set a default license under which models are uploaded when not specified?

In order to have a missing license added when it is missing, could you please advise on the standard way to proceed?
- Should an issue be created and tagging the author of the model asking for the license?
- Should I contact the author directly without raising an issue?

The community now has contributed a large number of very useful models, but more transparency regarding licensing (or default license) would be great.

Below are a few models I would be very interested in getting license information for, but a more general approach would be very beneficial:
- https://huggingface.co/valhalla/longformer-base-4096-finetuned-squadv1 (@patil-suraj )
- https://huggingface.co/mrm8488/longformer-base-4096-finetuned-squadv2 (@mrm8488 )
- https://huggingface.co/mrm8488/mobilebert-uncased-finetuned-squadv2 (@mrm8488 )
- https://huggingface.co/mrm8488/mobilebert-finetuned-ner (@mrm8488 )
- https://huggingface.co/mrm8488/mobilebert-finetuned-pos (@mrm8488 )

Thank you,


Comments:
Pinging @julien-c 

> I wanted to check if your conditions set a default license under which models are uploaded when not specified?

No, that's really the model author's call. But we will try to make it easier/more straightforward for a user to pick one in the future.

> In order to have a missing license added when it is missing, could you please advise on the standard way to proceed?

A GH issue is fine I think, otherwise a thread on [discuss.huggingface.co](https://discuss.huggingface.co) would work well too.

Thank you for the clarification!

If no license is available do we use the model for commercial use?
Fix post gemma merge
Author: ArthurZucker
Pull request: yes

# What does this PR do?
cc @LysandreJik 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31660). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix float out of range in owlvit and owlv2 when using FP16 or lower precision
Author: aliencaocao
Pull request: yes

# What does this PR do?
As discussed in https://github.com/huggingface/transformers/pull/31342#issuecomment-2192011957

The hard coded `-1e6` value used to mask logits is not in range for FP16 which causes a `value cannot be converted to type at::Half without overflow` error. We replace that with the min value of whichever dtype the logits are in.

Did a search and only the `owlvit` and `owlv2` models are using this, however, we should take note if any future model implementation hardcodes a similar value. There exists a couple more models using -1e7, -1e8, -1e9 and -1e10, but most of them are TensorFlow models, and the rest are either deprecated or language models. Since there has been no issue reports on this so far, I think it is properly handled somewhere so I won't change them.

Tests on FP32 showed no numerical difference in logits up to 1e-4 (what the test uses).

The reason of not using `float('-inf')` is that it can be 20% (RTX 3080Ti) to 3x (H100 SXM5) slower than using a predefined value for some reason.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts 


Comments:
@amyeroberts Anything else to do before it can be merged? It would fix the last failing test for https://github.com/huggingface/transformers/pull/31342

Nope! I'll merge now. Thanks for pinging and reminding me, and sorry for the delay!
[docs] Llama3
Author: stevhliu
Pull request: yes

Feedback from @gary149 about moving the `Quick usage` section to the top so it is faster for users to get started (also aligns with #31289).

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31662). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add Mixture of Tokens model
Author: chedatomasz
Pull request: no
Labels: name: New model

### Model description

Mixture of Tokens is a new architecture / technique proposed in [Mixture of Tokens: Efficient LLMs through Cross-Example Aggregation](https://arxiv.org/abs/2310.15961) and accompanying [blog](https://llm-random.github.io/posts/mixture_of_tokens/) by Szymon Antoniak, Sebastian Jaszczur et al.

It builds on expert-choice MoE, aggregating across sequences in a batch rather than positions in a sequence, and doing so in a continuous fashion. This full differentiability is its main advantage, bringing training stability and even expert utilization.

In collaboration with the authors, we (me + 3 others) would like to add a PyTorch implementation matching the architecture from the paper to HF transformers and later publish corresponding checkpoints. We believe this will make it significantly easier for the community to experiment with this approach, as the original implementation is quite dense and contained in an active research repo.

We believe a good approach is to start from the GPT2 HF model. We will have the assistance of the original authors for making sure the details match.

Please advise:
1.  If you have any general suggestions at this stage
2. What kinds of tests you would like to see in the finalized implementation for this case, where the exact snapshot corresponding to the paper's implementation and the checkpoints were not  previously published.
3.  If you have general suggestions regarding contributing methods that are potentially applicable to multiple base models  (like MoE and MoT).

As we understand, the next step is for us to create a template with https://github.com/huggingface/transformers/tree/main/templates/adding_a_new_model and get coding.



### Open source status

- [X] The model implementation is available
- [ ] The model weights are available

### Provide useful links for the implementation

https://github.com/llm-random/llm-random

https://github.com/sebastianjaszczur

https://llm-random.github.io/posts/mixture_of_tokens/



Comments:
The model is now available to try at https://huggingface.co/jaszczur/mixture_of_tokens
fix HF conversion for older Llama versions
Author: crackalamoo
Pull request: yes

# What does this PR do?

Fixes #30734 and #30723. Also fixes an additional error I was having on Linux, `OSError: [Errno 39] Directory not empty: 'llama-3-8b-hf/tmp'`.

An update adding Llama 3 to the file `convert_llama_weights_to_hf.py` involved a change in the dimension passed to the `permute` function. The new dimension (`dim1=dim // num_local_key_value_heads`) works for Llama 3, but breaks for older versions, which had simply used `dim1=dim`. Therefore this PR checks the version and passes the appropriate paramter.

Additionally, this PR ignores errors when deleting the `tmp_model_path` directory in case the directory is not empty, since it is temporary anyways and should still be deleted.

Tagging @ArthurZucker: I saw you were mentioned on some issues related to this, so let me know if this is a good fix. Thanks!

Comments:
Hi Arthur,

For my case, I need to install transformers==4.29.1. That solves my problem because it uses an earlier version of LLaMA weights.

Thanks a lot,

Yong
________________________________
From: Arthur ***@***.***>
Sent: Tuesday, June 25, 2024 11:08 AM
To: huggingface/transformers ***@***.***>
Cc: Wei, Yong ***@***.***>; Comment ***@***.***>
Subject: [EXTERNAL] Re: [huggingface/transformers] fix HF conversion for older Llama versions (PR #31554)

***This email is from an external source. Only open links and attachments from a Trusted Sender.***

@ArthurZucker commented on this pull request.

Hey! I open this: #31591<https://github.com/huggingface/transformers/pull/31591> for simplifcations, but will include your changes and add you as co author!

—
Reply to this email directly, view it on GitHub<https://github.com/huggingface/transformers/pull/31554#pullrequestreview-2138985835>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AVOENZMAKZKJNT3JJ3RDZATZJGBYRAVCNFSM6AAAAABJX4PU3WVHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDCMZYHE4DKOBTGU>.
You are receiving this because you commented.Message ID: ***@***.***>


The PR I link should help you, it also supports older version