noamaanMulla-03/transformers-issues huggingface dataset:
Implementation Issue of Phi3SuScaledRotaryEmbedding
Author: ryan-minato
Pull request: no
Labels: name: Feature request

### Feature request

Improving the Implementation of Phi3SuScaledRotaryEmbedding to Reduce Unnecessary Computation

I'm not entirely sure if there is a deeper meaning to the implementation here. It seems that the inv_freq could be completely initialized in the `__init__ `method as `long_inv_freq `and `short_inv_freq`. 
The only parameter used here is the part that gets the device.

https://github.com/huggingface/transformers/blob/25245ec26dc29bcf6102e1b4ddd0dfd02e720cf5/src/transformers/models/phi3/modeling_phi3.py#L131-L137

### Motivation

Fixing this code can eliminate a redundant computation. Moreover, self.inv_freq used here doesn't seem appropriate to be a attribute.

I tested the modified version locally. Although I couldn't test with long contexts due to my device's performance, the new implementation doesn't seem to show any issues with short sentences.

### Your contribution

I can fix it if this isn't intentional.

Comments:
cc @ArthurZucker @gante 

PR reviewed! 🤗
Add docs on zeroshot image classification prompt templates
Author: aliencaocao
Pull request: yes
Labels: name: Documentation

name: Vision

# What does this PR do?

Adds documentation on prompt templates for Zero Shot Image Classification tasks so that the output results match pipelines and are also correct. Not applying the template can lead to greatly worsened accuracy.

<!-- Remove if not applicable -->

Fixes #30951 


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@amyeroberts


Comments:
i have a PR that adds support for siglip training coming up so would be great if this could be merged asap so the docs don't merge conflict

> i have a PR that adds support for siglip training coming up so would be great if this could be merged asap so the docs don't merge conflict

@aliencaocao I appreciate that you want to have PRs merged in as soon as possible (we do too!) but managing conflicts for other PRs isn't something that we will consider, unless it's also for transformers something we want to merge in and linked to. 

@amyeroberts heads up i've made the changes already

Merged your suggestions

@aliencaocao Can you try rebasing to include the latest changes upstream? This should resolve some of the issues on the CI runs 

@amyeroberts hi somehow the workflows arent getting triggered?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31343). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@aliencaocao We've had a few issues with tests triggering on circle CI today. I'm not 100% sure of the cause, but other PRs which have blocked workflows are now working. Could you try triggering a new run? `git commit --allow-empty -m "Trigger CI"`

@amyeroberts its ready now

Thanks for updating the docs and your patience with the CI!
🚨 FLAVA: Remove double softmax
Author: amyeroberts
Pull request: yes

# What does this PR do?

Removes duplicate softmax application in FLAVA attention. Likely to have a small change on the outputs but flagging with 🚨 as it will change a bit. 

Applies same change as in #30198 which hasn't had recent activity. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31322). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix circular reference issue in CLIPTokenizerFast
Author: dhaivat1729
Pull request: yes

# What does this PR do?

In CLIPTokenizerFast, `backend_tokenizer.decode` references to new_decode_method method, which in-turn uses `CLIPTokenizerFast` object reference, causing circular reference and thus, failing to release CLIPTokenizerFast after garbage collection.

Fixes #30930


## Who can review?
@ArthurZucker 


Comments:
Would usually ask to add a test, but this is cumbersome to test so let's just merge!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31075). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@ArthurZucker 
> Would usually ask to add a test, but this is cumbersome to test so let's just merge!

Could you tell me what kind of tests maybe required? I would love to contribute but need some directions as I am new here. I am ready to put efforts into this, let me know if you want to add more to make this change more robust. 

Don't worry! We usually want to have a test that check the behaviour we fixed was actually fixed. But this is alright in this case testing is hard!
LLama3 won't stop when provide eos_token_id
Author: lucasjinreal
Pull request: no
Labels: name: Generation

I have a model genereate config like this:

```
output_ids = model.generate(
                input_ids,
                images=images_tensor,
                do_sample=False,
                temperature=0.2,
                max_new_tokens=1024,
                streamer=streamer,
                use_cache=True,
                # cache_position=None,
                pad_token_id=tokenizer.eos_token_id,
                eos_token_id=conv.stop_token_ids,
                stopping_criteria=[stopping_criteria],
            )
```

the pad token id is 
 it's wont stop when 
 got which needed for each assitatn message stops.

So I provide that id in eos_token_id, but still, it doesn't respect it. how to resolve it?

Comments:
Hey, can your provide a more complete code to reproduce it (e.g. model_id, what's "conv.stop_token_ids")?

In general providing in `eos_token_id` an int or a list of int (when two or more tokens can be `eos`) should stop generation. Make sure that the generated text contains one of the provided `eos_token_ids`, because sometimes the same string can be mapped to another sequence of `token_ids`

I get a similar issue. The reason for this might be that part of `
` is combined with other output and becomes two tokens. For example,  `Over.
` is tokenized to `Over.<|e` and `ot_id|>`. Is there a way to solve this issue? 

Or does the tokenizer in Transformers support first tokenizing the message, then adding the token id of 
?

@zucchini-nlp 

@gasvn yeah, that could be the reason. If you want to fix the generation and stop at "<eot_id>", you can use [StopStringCriteria](https://huggingface.co/docs/transformers/en/internal/generation_utils#transformers.StopStringCriteria) and pass eos as a string. Should be working.

For skipping special tokens at tokenizer level, I believe you can use ["add_special_tokens" arg](https://huggingface.co/docs/transformers/main_classes/tokenizer#transformers.PreTrainedTokenizer.__call__) when calling tokenizer, and manually append EOS and BOS later. Ping @ArthurZucker and @itazap 

If you are training the base model, it is expected that the eot token is not special nor added because the base model was not trained with it. You need to 1st add it, then make sure you set `add_eos_token = True` and set `add_special_tokens = True` when encoding to automatically add your new eos token. Set it to false to skip the addition of bos en eos
Fix `wandb` integration with `SetFit` model
Author: timothepearce
Pull request: yes

# What does this PR do?
This PR fixes [an error popping up](https://github.com/huggingface/setfit/issues/498) while training a `SetFitModel`while reporting to `wandb`.

To fix this issue, we could also subclass and override the `setup` method specifically in the `setfit` repository, but that would be a much bigger job. @tomaarsen if think you're the one who should decide on that.

Fixes # (issue)
https://github.com/huggingface/setfit/issues/498

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@tomaarsen @ArthurZucker 

Comments:
@ArthurZucker Done!

Just need the CIs to go Green 😉 you can rebase on main and run `make fixup`! 🤗 

@ArthurZucker That's what I did, but `run_tests`and `ci/circleci: tests_tf` are still failing, as it on the main branch. 😄

Just re-triggered them! 

the test seems unrelated so I think we can merge!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30021). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
ALBEF: Align Before Fuse
Author: ggoggam
Pull request: no
Labels: name: New model

### Model description

Align Before Fuse (ALBEF) is a vision-language (VL) model that showed competitive results in numerous VL tasks such as image-text retrieval, visual question answering, visual entailment, and visual grounding. 

The authors propose to use text encoder (BERT's first half layers) and image encoder (ViT) to create an aligned representation for respective modality before fusing them together with a multi-modal encoder (BERT's second half layers). The model is trained on multi-modal representation tasks and momentum distillation to achieve state-of-the-art results in VL tasks.

As multi-modal models are gaining more attention in academia/industry, I think ALBEF could be a nice addition to the transformers library.



### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

- There are an official implementation and pre-trained/fine-tuned weights by the authors at this [repo](https://github.com/salesforce/ALBEF)
- Link to the [paper](https://arxiv.org/abs/2107.07651)

Comments:
@jkgrad What is the state of this issue? If no one is working on this, I would like to implement it.

Hey @DanielFLevine, we'd love for you to try and contribute that model! 

cc @NielsRogge who can help out once he's back from leave :)

@LysandreJik @NielsRogge Great! I've already started looking over the authors' code. Will reach out with any questions.

Is there still interest for this?

Same question

@DanielFLevine-zz  - any updates on the model port?

any progress on this?
will this be a model or just a tokenizer/embeddings ? technically a sentence trancformer ?
Author: spydaz
Pull request: no
Labels: name: Feature request

### Feature request

for the text component : will this config be abe to take a languge model as a pretrained config ? 

ie when creating a vision/text decoder model you can specify the tet model component ( ie the mistral model ) ? 
as the mistral is a decoder only model .... maybe it would need corss attention to connect ?
will the text componant be able to be used with the standard AutoModelForCausalLM ? <<< If so it would be prudent to be able to intialize a model with the pretrained components ... ie a lnaguge model and a whisper and a clip model and a xclip ? or to the effect of ... enableing for the genration of modelif these (bodys) ... ie collection of tensors whcih can be read by the imagebind library  transfromer... instead of a simple tokenizer .... it would then be able to utilize the tokenizer from the llm or combine the tokenizers into a single model by adding the speicla tokens to the languge model base ? 

in truth this would enable the generation of even larger models and give the pretrained model a chance to find thier homes in the new body of a (imagebind) as it is the wrapper which makes the model ...(hence perhaps also including a feature to execute code on the fly and return the executing back to the model directly if enabled or ok byu the user...as this is the method as the model could generate its on code to execute in opython and return the result to the transformer to be utilized in the response it is generating .... so if the model is given functions as an input (it would also be able to run them in the back ground pre retuning the response) ... also given code by the user in chat it should be able to run the code in ipython behind the scenes and return the response (not on the system directoy(hence havign hiddon ipython repl) ..(it would still have to make the calls)

### Motivation

rags internally ! 
usablity ! of the model wrapper to understand that each component could be trained seperate or even within the model itself !

as well as fitting with anything in and wnything out principle (media wise) (code as an input (function)) 

### Your contribution

i really would like to see this project get forwards as its the same thing i was working towards but i have fiath you will succeed ... if not i will stillc arry on my path !

Comments:
AutoModelForCausalLM.from_pretrained silently fails
Author: gpetters-amd
Pull request: no

### System Info

Running `AutoModelForCausalLM.from_pretrained` as in the following script silently fails on Windows 10. I've attached a minimal reproducer and the pip freeze of the environment I'm using.

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. `pip install -r freeze.txt`
2. `python repro.py`

### Expected behavior

The script should print "Done."

Comments:
[Repro files are here.](https://gist.github.com/gpetters-amd/833bd956af4ccc18dca90d956411b7c3)

Hi @gpetters-amd , it's happening because your RAM goes out of memory while instantiating the pretrained model
Please change your code to the following in `repro.py` :
```
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
    "Trelis/Llama-2-7b-chat-hf-function-calling-v2", token=None, config=None,low_cpu_mem_usage=True,offload_state_dict=True
)

print("Done.")
```
`low_cpu_mem_usage` : **'it will load the model using ~1x model size CPU memory'**
`offload_state_dict` :  **'it will temporarily offload the CPU state dict to hard drive and will prevent getting out of RAM'**

Cheers!

> Hi @gpetters-amd , it's happening because your RAM goes out of memory while instantiating the pretrained model Please change your code to the following in `repro.py` :
> 
> ```
> from transformers import AutoModelForCausalLM
> model = AutoModelForCausalLM.from_pretrained(
>     "Trelis/Llama-2-7b-chat-hf-function-calling-v2", token=None, config=None,low_cpu_mem_usage=True,offload_state_dict=True
> )
> 
> print("Done.")
> ```
> 
> `low_cpu_mem_usage` : **'it will load the model using ~1x model size CPU memory'** `offload_state_dict` : **'it will temporarily offload the CPU state dict to hard drive and will prevent getting out of RAM'**
> 
> Cheers!

Oh, you're right. Thanks!
Fix a teeny-tiny typo in `tokenization_utils_base.py`'s docstring
Author: sadra-barikbin
Pull request: yes

Hi there!

To fix a typo in `tokenization_utils_base.py::PreTrainedTokenizerBase::truncate_sequences`'s docstring.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31510). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
GGUF: Fix llama 3 GGUF
Author: younesbelkada
Pull request: yes

# What does this PR do?

Replaces: https://github.com/huggingface/transformers/pull/31215/ 
Fixes: https://github.com/huggingface/transformers/pull/30391#issuecomment-2145637564

Currently it is not possible to load Llama-3 GGUF models due to the fact that the Llama3 tokenizer is slightly different from the previous Llama models. 

A way to detect that we are having a llama-3 gguf model is to check for the attribute `tokenizer.model` (registered as `tokenizer_type` in `proto`) of the GGUF file (see an example below taken from [this checkpoint](https://huggingface.co/NousResearch/Meta-Llama-3-8B-GGUF/)). 

![Screenshot 2024-06-10 at 17 39 26](https://github.com/huggingface/transformers/assets/49240599/da887bbf-697d-4ef7-bec8-64c68df20779)

Firstly, the GGUF file directly contains the merges attributes, in that case, `scores` is not present anymore so creating a dummy scores array is sufficient and avoids errors. 

Secondly, I addressed the case where `unkonwn_token_id` is registered within the proto object ( seems to be the case for some Llama2 checkpoints)

In addition to that, note that llama3 uses different special tokens that are different from the default special tokens of LlamaTokenizer (e.g. it uses `<begin_of_text>` instead of `
`). This can be fixed by passing the correct special tokens to the tokenizers init method - therefore we need a logic to pass these new args / kwargs to that method. I propose to create a new attribute `additional_kwargs` inside the tokenizer converter and pass that along to the conversion logic. Moreover, I made sure that the special tokens handling is more "universal" (in the past we were hardcoding everything)

+ For the decoding process (thanks @ArthurZucker for the offline explanation), one needs to use `decoders.ByteLevel` to properly decode the generated tokens as spaces are encoded with the character `Ġ` in the Llama3 tokenizer - this is not the case for Llama 1 & 2 tokenizers.

```bash
In [19]: tok.encode("Hello I am new to this forum").tokens
Out[19]: ['
', 'Hello', 'ĠI', 'Ġam', 'Ġnew', 'Ġto', 'Ġthis', 'Ġforum']
```

Finally I added a new test that uses a Llama3 checkpoint and I made sure previous tests all pass


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31358). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
RWKV: enable generation tests
Author: gante
Pull request: yes

# What does this PR do?

Follow-up to #31478 

Enables generative tests in RWKV, making use of the `_is_stateful` model flag and `has_attentions` test flag.

RWKV has outputs with unique shapes, a few tests were skipped. Some of those unique shapes have custom tests (e.g. attention outputs). Others don't (hidden states with `use_cache=True`), but given the model's low usage I don't think it's worth diving in.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31490). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts done 🤗
Add PLBartTokenizerFast
Author: gchhablani
Pull request: yes

This PR adds fast tokenizer for PLBart. It is currently a work-in-progress.

@patil-suraj @sgugger @LysandreJik @patrickvonplaten 

### To-do:
- [ ] Add missing classes convert_slow_tokenizer
- [ ] Convert and add the tokenizer files to all the model checkpoints
- [ ] Fix the tests

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_16551). All of your documentation changes will be reflected on that endpoint.

@sgugger I am actually facing an issue for the converter. The `PLBartTokenizer` accepts an argument called `language_codes` based on which the language codes for the tokenizer are decider. 

The converter only takes one list of language codes. 

Should I modify the method which is used in `PreTrainedTokenizerFast` to allow for adding extra arguments like `language_codes`? What is the best way of going about this according to you?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Unstale

Hey @gchhablani ! Let us know if you need any help with this :) 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Any updates regarding the PLBART fast tokenizer?
[run-slow] cohere
Author: ydshieh
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Not sure why, but those 2 tests are passing now.
(even if I checkout to one commit before #31263)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31517). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Revive Nightly/Past CI
Author: ydshieh
Pull request: yes

# What does this PR do?

The CI workflow files have changed a lot since this year:

- #28773 to allow more than 256 jobs in a workflow run
- #30012  allow each job to be called independently and send the report to different channels earlier
- a few more smaller changes regarding CI report: #31013, #30968

The nightly/past CI workflow are currently not working due to the above changes. This PR revives these 2 workflow.

**For Past CI**

- Previously, we run all torch/tf version in a single workflow run, which is too large and we can't access the workflow run page (`This page is taking too long to load`). This PR changes it to run in different scheduled time and using an index to determine which torch/tf version to run. However, each run will run against a different `transformers` commit which is not ideal, but acceptable for now.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31159). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Awesome! Let me know when you'd like for me to review this

> Awesome! Let me know when you'd like for me to review this

Hi! It's ready for a review, but I could also walk you through a bit the changes next Tuesday if that would help and save time.

(I triggered some runs to make sure daily/nightly/past CIs will run, but maybe let me to perform a final check before merge on my side)



~~I forgot to update `utils/notification_service_quantization.py` (tiny changes required too). I will update it.~~

Updated