noamaanMulla-03/transformers-issues huggingface dataset:
Add a community notebook for fine-tuning with QLoRA, PEFT, and MLflow
Author: daniellok-db
Pull request: yes

# What does this PR do?

This PR adds a community notebook tutorial for fine-tuning an open-source LLM using QLoRA and PEFT, using MLflow for experiment tracking.

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->

cc @stevhliu as the recommended reviewer for documentation

Comments:
Fix: StaticCache & `inputs_embeds`
Author: zucchini-nlp
Pull request: yes

# What does this PR do?

Fixes #32911. Enables generation with Static Cache and inputs embeds, previously it was failing due to incorrect calculation of `max_cache_length`

Added a test for that and added tests for `Gemma2ForCausalLM`. Some things to note:
- Gemma2 doesn't support `StaticCache`. It can with some small changes but imo we shouldn't
- Static shape cache classes have no support for contrastive search, dola, low-memory generation and assisted decoding. So these tests are all skipped in Gemma2. I think if we want to enable the, it should go on another PR for upgrading static cache classes

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32932). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi, run into similar errors as in https://github.com/huggingface/transformers/issues/32911, will this PR get merged?

Yes, merging now, should be ready
Is it possible to make wasm support all models in huggingface?
Author: wa008
Pull request: no
Labels: name: Feature request

### Feature request

As title
Now, there are several model that support wasm.
1. [whisper.wasm](https://github.com/ggerganov/whisper.cpp/tree/master/examples/whisper.wasm), which can convert audio to text.
2. [secret-llama](https://github.com/abi/secret-llama): text-to-text LLM, realize by web-gpu
3. web-llm: text-to-text LLM
If it is possible to make a project to support wasm for all huggingface models, this can definitely accelerate the development of LLM.

### Motivation

Wasm have below advantages

1. run locally, no server or extran machine needed, which can help user to try new model without any cost.
2. no any privacy concerns

### Your contribution

I can help to develop related project or function.
I know it will be a huge project,I can not do that by myself.

Comments:
The best suited to answer you is probably @xenova :)

Hi @wa008! I responded to a similar question [here](https://huggingface.co/posts/victor/964839563451127#66d7f103559b4669c53df7e6) (maybe you?), so let me copy my response here:

> We have Transformers.js, the JavaScript/WASM/WebGPU port of the python library, which supports ~100 different architectures.
> - Docs: https://huggingface.co/docs/transformers.js
> - Repo: http://github.com/xenova/transformers.js
> 
> Is that the kind of thing you're looking for? :)

> Hi @wa008! I responded to a similar question [here](https://huggingface.co/posts/victor/964839563451127#66d7f103559b4669c53df7e6) (maybe you?), so let me copy my response here:
> 
> > We have Transformers.js, the JavaScript/WASM/WebGPU port of the python library, which supports ~100 different architectures.
> > 
> > * Docs: https://huggingface.co/docs/transformers.js
> > * Repo: [http://github.com/xenova/transformers.js](https://github.com/xenova/transformers.js)
> > 
> > Is that the kind of thing you're looking for? :)

Yes, this project is great! Thanks a lot.
Whisper Alignment Heads calculation for custom model
Author: DavraYoung
Pull request: no

### Feature request

Token timestamps work great on the pretrained model, but once the model is finetuned, token timestamp are no more correct.
I tried to dig deeper and found that the token timestamps calculation is done using cross attention heads with some dynamic time warping calculation over token sequence.

it would be good to have some discussion or script that allow to select the best alignment heads for any whisper model.


What were your approach for selecting those alignment heads?

### Motivation

Token timestamps, long form transcription, hallucination detection etc.

### Your contribution

I have tried calculating best alignment heads using this approach:

0. Select 100 good enough sample audios.
1. Use some CTC model for per word timestamp calculation, calculate per word timestamps
2. Run whisper with token timestamps and get timestamps considering each attention head separately
3. Consider the time difference for each attention head with the one from CTC model (for matching token transcriptions)
4. Select the attention head with best accuracy
5. Repeat from step 1, using the selected head, calculate all other heads that in combination with current heads can improve the accuracy

I was able to achieve average 200ms accuracy for token timestamps. Accuracy on best 80% is 40ms. The model fails only on some cases having 2000-4000ms shift, which can be detected using some scripts.

Comments:
@sanchit-gandhi hi, have you tried this with distilled model?

Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!

Thanks!

Hey @DavraYoung! Indeed - as @ArthurZucker mentioned the forum is the best place to ask such questions. Linking some resources that may be of help for reference for your question to get you going:
* Answer from Whisper author: https://github.com/openai/whisper/discussions/1388#discussioncomment-7568428
* Thread on the HF Hub: https://huggingface.co/distil-whisper/distil-small.en/discussions/7

Hey @DavraYoung - sorry to dig up this old issue! I was wondering if you had a script for running the time alignment according to your algorithm? I do think it's indeed more systematic than manually inspecting the heads. I'd love to give it a go for Distil-Whisper and work with you to add this as a resource.

checkout this repo for the best individual alignment heads of the large v2 and v3 variants.

https://github.com/nyrahealth/CrisperWhisper/blob/develop/run_experiments/experiments/head_results.json

The Repo also contains useful code to calculate these head results for other models as long as you are in the posession of a dataset with high quality timestamps like timit ( https://paperswithcode.com/dataset/timit). 


https://github.com/nyrahealth/CrisperWhisper

Let me know if i can be of further help.
[Zero-shot image classification pipeline] Remove tokenizer_kwargs
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR is a follow-up of #29261, namely the `tokenizer_kwargs` argument is unnecessary, one can just update the `model_input_names` attribute of the tokenizer.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33174). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> It is both a breaking change in terms of the pipeline's functionality and will break for all existing Blip2 checkpoints saved with a bert tokenizer

The Blip2 checkpoints compatible with this pipeline are the following:
* https://huggingface.co/Salesforce/blip2-itm-vit-g
* https://huggingface.co/Salesforce/blip2-itm-vit-g-coco

The pipeline was updated in a just-merged PR (which also pushed those 2 checkpoints). My point would be to just set the `model_input_names` of the tokenizer of those 2 repos appropriately which would ensure the `tokenizer_kwargs` isn't needed - I also don't see it present in any other pipeline

OK, as `tokenizer_kwargs` were just added for a model on `main` then we can remove. Could you open PRs on the BLIP2 checkpoints to update to make sure they are compatible? 

It seems the `model_input_names` attribute doesn't get serialized in `tokenizer_config.json` (as it's a class attribute), cc @ArthurZucker. So I'm not sure it's possible to remove this.

@NielsRogge In this case, we should just keep the `tokenizer_kwargs` in the pipeline, as they're a fairly standard input anyways. The issue regarding having to pass `{return_token_ids: "False"}` for BLIP will hopefully be something we can resolve once we can load processors in pipelines - #32514, as `tokenizer.return_token_ids` will be automatically set. 

What should be done is document that we need to pass in `{return_token_ids: "False"}` to use these blip2 checkpoints in the pipeline 

I think `model_input_names` should be serialized! (some model like mamba use GPTNeoXTokenizerFast, but don't use token_type_ids for example)
Docs: add more cross-references to the KV cache docs
Author: gante
Pull request: yes

# What does this PR do?

@zucchini-nlp wrote a great guide for our KV caches. This PR adds more cross-references to that guide in relevant places. 

These references were added because: 
1. a docstring can't fully explain what a cache can be 
2. users have noticed that our cache classes are underdocumented/not easy to find (e.g. #33270)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33323). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add static cache support for Whisper
Author: mobicham
Pull request: no
Labels: name: Feature request

name: Audio

### Feature request

Would be great to have static cache support for Whisper to make it faster with `torch.compile`. Currently, the `generate()` function doesn't support `cache_implementation="static"` for Whisper. 

### Motivation

Static cache with torch.compile can make generation much faster.

### Your contribution

 Static cache is already supported for LLMs and we see great speed-up.

Comments:
cc @sanchit-gandhi 

Let me try, I think I can make it, just need to patch https://github.com/huggingface/transformers/blob/main/src/transformers/models/whisper/modeling_whisper.py#L313 like llama model and pass `cache_position` should be ok

@huseinzol05 great, thanks ! I think you also need to make sure the model supports initializing the static cache via `_setup_cache`:
```Python
from transformers import StaticCache
model._setup_cache(StaticCache, batch_size, max_cache_len=max_cache_length)
```

I got hit by https://github.com/pytorch/pytorch/issues/123592 at https://github.com/huggingface/transformers/blob/main/src/transformers/models/whisper/modeling_whisper.py#L230, but the static cache is already working without torch compile from my local, arange should solved the problem

Maybe you can use `arange` instead like here: https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L964-L966

Anything dynamic not possible, feed `position_ids` solved the problem, just like `cache_position`, i will push the initial later, so you can verify, the speedz is good

Great :+1: ! But that `arange` works well in Llama with fullgraph torch compile.

https://github.com/huggingface/transformers/pull/30760

Compiled static cache able to achieve 186.26it/s while non-compiled got 150.20it/s

Closing as this is fixed: #31166 and #31772
Test that #33276 works well
Author: ArthurZucker
Pull request: yes

Just makes sure that #33276 is working as expected: for a simple diff only have 1 parallel node

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33279). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
support loading model without config.json file
Author: itazap
Pull request: yes

We already have support for loading a fast tokenizer with a `tokenizer.model` file only. However, we still require `config.json` to exist in the model folder (on hub or locally), even though it is not required to load from the model file. This feature will remove this dependency on `config.json`, allowing users to load from only a `tokenizer.model` file. 

https://github.com/huggingface/transformers/pull/31656 (TikToken file support) depends on this

TESTS:
- updated stale tests 
- updated [existing test](https://github.com/huggingface/transformers/blob/30b673d83d906a3069d25f89eb92bc1bd5cf1690/tests/utils/test_configuration_utils.py#L249) to work without config file
- added test in llama to load with only `tokenizer.model `file

Reviewer:
@ArthurZucker 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32356). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[Help] Correct Way to do Simple Model Constraints
Author: JamesKCS
Pull request: no

I am trying to figure out the correct way to apply simple model output constraints with transformers.  As an example, let's consider how to constrain the output to a non-negative int with an unknown number of digits (decided by the model).  For example, it could output `11`, or `0`, or `307`, but **not** `307.` or `the answer is 307`.

I've read https://huggingface.co/blog/constrained-beam-search; this is a great resource, but it is solving a much more challenging problem than the kind of simple model constraints that I'm looking for, and it doesn't tell us how to solve this easier problem.  I'm also aware of the many 3rd party packages that accomplish this, but I want to know the correct way of doing it just using the Transformers package.

Could someone point me in the right direction (perhaps there's a blog post that I missed), or give me a code snippet of the best way to accomplish this?  Perhaps I should be using something from [beam_constraints](https://github.com/huggingface/transformers/blob/main/src/transformers/generation/beam_constraints.py) or [tf_logits_process](https://github.com/huggingface/transformers/blob/main/src/transformers/generation/tf_logits_process.py)?  Thank you.

Comments:
We don't support controlled LLM generation via grammar in Transformers directly, but there are many awesome external libraries for that. You might be interested in https://github.com/outlines-dev/outlines or https://github.com/uiuc-focal-lab/syncode, as an example 🤗 

Thank you for the reply. 

>via grammar

I'm not looking for anything this sophisticated. For the above example, I'd be just looking to set the probabilities of all tokens *except* 0-9 and the EOS token to 0 (which should result in it the model being forced to output an int).  If the [more sophisticated constrained beam search](https://huggingface.co/blog/constrained-beam-search) is a possibility without external libraries, then surely this simpler case is also doable without external libraries?  Like I mentioned, I need to do this without external libraries.  Thank you in advance.

@JamesKCS in that case you can write your own LogitsProcessor that will set all probabilities to `-inf`, please refer to the docs https://huggingface.co/docs/transformers/en/internal/generation_utils#logitsprocessor. You have to write something similar that suits your use case.

If you have any questions please open a discussion in the [forum](https://discuss.huggingface.co/latest), we try to reserve GH for bugs and features 🤗 

Thank you, sorry for posting this in the wrong place, I'll close this (and maybe update it later with a MWE for future searchers).
sdpa for bert should support 4D  attention mask.
Author: Leoyzen
Pull request: no
Labels: name: Good First Issue

name: Feature request

### Feature request


Currently this is only possible with 2D-mask when sdap is enabled.
```python
# modeling bert
        # Expand the attention mask
        if use_sdpa_attention_masks:
            # Expand the attention mask for SDPA.
            # [bsz, seq_len] -> [bsz, 1, seq_len, seq_len]
            if self.config.is_decoder:
                extended_attention_mask = _prepare_4d_causal_attention_mask_for_sdpa(
                    attention_mask,
                    input_shape,
                    embedding_output,
                    past_key_values_length,
                )
            else:
                extended_attention_mask = _prepare_4d_attention_mask_for_sdpa(
                    attention_mask, embedding_output.dtype, tgt_len=seq_length
                )
            #  print(f"sdpa attn mask:{embedding_output.dtype=} {extended_attention_mask.sum(axis=(2,3))=}")
        else:
            # We can provide a self-attention mask of dimensions [batch_size, from_seq_length, to_seq_length]
            # ourselves in which case we just need to make it broadcastable to all heads.
            extended_attention_mask = self.get_extended_attention_mask(attention_mask, input_shape)
```

### Motivation

4D attention mask is widely used such as padding bert.

We construct 4D attention mask before the model forward and pass it to the forward call.

I think it is easy to exend the current implement of _expand_mask to support 3D/4D attention mask.

```python
    @staticmethod
    def _expand_mask(mask: torch.Tensor, dtype: torch.dtype, tgt_len: Optional[int] = None):
        """
        Expands attention_mask from `[bsz, seq_len]` to `[bsz, 1, tgt_seq_len, src_seq_len]`.
        """
        bsz, *_, src_len = mask.size()
        tgt_len = tgt_len if tgt_len is not None else src_len

        if mask.ndim == 2:
            expanded_mask = mask[:, None, None, :].expand(bsz, 1, tgt_len, src_len).to(dtype)
        elif mask.ndim == 3:
            expanded_mask = mask.unsqueeze(1).to(dtype)
        elif mask.ndim == 4:
            expanded_mask = mask.to(dtype)

        inverted_mask = 1.0 - expanded_mask
        #  print(f"{dtype=} {torch.finfo(dtype).min=} {expanded_mask.bool().sum((1,2,3))=}")

        return inverted_mask.masked_fill(inverted_mask.to(torch.bool), torch.finfo(dtype).min)
```

### Your contribution

I can do PR and contribution.

Comments:
cc @ArthurZucker @fxmarty 

Sounds good! Feel free to open a PR for a fix!
ImportError: attempted relative import with no known parent package
Author: BaneTrifa
Pull request: no
Labels: name: bug

### System Info

transfromers version: 4.44.2
pytorch version: 2.4.1+cu118

### Who can help?

@gante @ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I'm trying to run the official BERT test (**tests/models/bert/test_modeling_bert.py**) for the purpose of debugging my own code. However, when I run the test, I get the following error:
```
Traceback (most recent call last):
  File "/home/huggingface/transformers/tests/models/bert/test_modeling_bert.py", line 30, in <module>
    from ...generation.test_utils import GenerationTesterMixin
ImportError: attempted relative import with no known parent package
```

I've seen a few solutions for similar issues, but none of them solved my problem.

### Expected behavior

I expect to run official BERT tests without import errors.

Comments:
Hi @BaneTrifa 👋 Thank you for opening this issue

You should be able to run tests if you are in the transformers root folder (e.g. you can call `py.test tests/models/bert/test_modeling_bert.py`). Our tests use relative paths and have this assumption.

@gante It solved my problem. Thanks
Unrecognized keys in `rope_scaling` for 'rope_type'='dynamic': {'type'}
Author: DefTruth
Pull request: no
Labels: name: bug

### System Info

version 4.44.1


### Who can help?

@ArthurZucker 


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Unrecognized keys in `rope_scaling` for 'rope_type'='dynamic': {'type'}

### Expected behavior

no error, version 4.42.4 is OK, but 4.44.1 raise this error message, config:  
```json
"rope_scaling": {
    "factor": 3.0,
    "type": "dynamic"
  }
```

Comments:
There has been an update to how rope_scaling key is handled. When llama3.1 came out, transformers refactored RoPE logic https://github.com/huggingface/transformers/pull/32135 

Indeed, tho we did try to make things backward compatible. Do you have a reproducer? 

FWIW - I see it when loading a vicuna models.

```
>>> model = AutoModelForCausalLM.from_pretrained('lmsys/longchat-7b-v1.5-32k', do_sample=True)
Unrecognized keys in `rope_scaling` for 'rope_type'='linear': {'type'}
Loading checkpoint shards: 100%
>>> 
```

```
model = AutoModelForCausalLM.from_pretrained('lmsys/vicuna-7b-v1.5-16k', do_sample=True)
Unrecognized keys in `rope_scaling` for 'rope_type'='linear': {'type'}
Loading checkpoint shards:   100%
>>> 
```


Yep I can reproduce. cc @gante

Yup, can reproduce it too. It is a harmless warning (no impact on model usage), but opening a PR for it
RoPE: fix BC warning
Author: gante
Pull request: yes

# What does this PR do?

Fixes #32916 

A warning is thrown when there are unused keys in the rope config. When the old key `type` is present, we copy its contents to `rope_type`. We forgot to remove `type` from the unused keys check in this case, causing the warning to be thrown 🤗 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33331). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.