noamaanMulla-03/transformers-issues huggingface dataset:
Add HQQ model serialization support
Author: mobicham
Pull request: yes
Labels: name: Quantization

## What does this PR do?
The goal of this PR is to add support for saving/loading transformers models quantized via <a href="github.com/mobiusml/hqq/">HQQ</a>. 
Currently, it's not possible to directly save/load HQQ-quantized models via `model.save_pretrained`. The workaround is to use the `hqq` lib which offers serialization but without safetensors support.

The first thing to do was making the `HQQLinear.state_dict` compatible with safetensors: https://github.com/mobiusml/hqq/commit/74bbe01e51dcba34d6d2ab6ded22bedcdb89d2f7
Since safetensors only support `torch.Tensor` dtype and no nested dictionaries are allowed, I had to implement an encoding/decoding logic for the parameters, which works fine for the moment.

The second step was integrating the new `HQQLinear` state_dict loading with the transformers lib, which I found a bit tricky because of the following issues:  
- Some state_dict parameters are split across 2 safetensors files. So loading fails for that specific layer since I can only get the parameters from the first file. For now, I put a fake quantized layers so that loading doesn't break but you'll see a message.
- `dispatch_model` breaks with `HQQLinear` module, I am not sure what's the issue here. Before it reaches that function, the layer is already loaded from the state_dict. If I skip the dispatching function, I a get a model with meta device for all the layers except `HQQLinear`

Since this is a draft pull-request, I am commenting out some stuff while debugging, will fix that later (you'll see there's a `TODO @mobicham` comment so I can get back to it).

Will need to drop support for quantized scale/zero, so I will need to update the HqqConfig and the documentation, as well as add tests later.

Any help is highly appreciated , thank you!

## Who can review?
@SunMarc

## Example
```Python
import torch, gc
from transformers import AutoModelForCausalLM, AutoTokenizer, HqqConfig

quant_config  = HqqConfig(nbits=4, group_size=64, quant_zero=False, quant_scale=False, axis=1)

model = AutoModelForCausalLM.from_pretrained(
    'meta-llama/Meta-Llama-3-8B', 
    torch_dtype=torch.float16, 
    cache_dir='/nas/hicham/tmp/',
    device_map="cuda:0", 
    quantization_config=quant_config
)
 
#Test 
with torch.no_grad():
	out = model.forward(torch.zeros((1, 8), dtype=torch.int32, device='cuda:0'))


# Save
model.save_pretrained("llama3-hqq")
del model
torch.cuda.empty_cache(); gc.collect();

#Load
model_loaded = AutoModelForCausalLM.from_pretrained(
    'llama3-hqq', 
    torch_dtype=torch.float16, 
    cache_dir='.',
    device_map="cuda:0")

## Does't work here because of `dispatch_model()`

with torch.no_grad():
	out = model_loaded.forward(torch.zeros((1, 8), dtype=torch.int32, device='cuda:0'))
```


Comments:
Gentle ping @SunMarc
When the model outputs Chinese, the dimension of the attention map output by output_attentions is wrong
Author: zhaozhipeng1997
Pull request: no
Labels: name: Feature request

### Feature request

When outputting English, the last layer of attention map is (960, 960), and when outputting Chinese, the last layer of attention map is (1910,). How can this problem be solved?

### Motivation

When I use llama to output Chinese

### Your contribution

no

Comments:
Virus

Hey @zhaozhipeng1997, could you share a script of what you're trying to do and what you identify as being wrong? Thanks!
disable scheduled daily CI temporarily
Author: ydshieh
Pull request: yes

# What does this PR do?

We have trouble with daily CI runs. 

It seems our daily CI run is not able to finish in 24 hours, and once the next day's run is triggered, things get worse.
It's not very clear what happens, but I see

- qwen2_moe --> can't finish in 6 hours
- gemma2 --> finished in 3 hours

In order to see what would a run go  (especially if it could even finish and/or how long) **without a next run being triggered**, let's disable the scheduled daily CI on `main`  for  1 or 2 days, and manually a run.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33136). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Shouldn't we skip qwen2 moe instead as a quick fix?
CI: fix `efficientnet` pipeline timeout and prevent future similar issues due to large image size
Author: gante
Pull request: yes

# What does this PR do?

This PR:
- Should fix the timeout we've been seeing in the efficientnet pipeline
- Adds exceptions to prevent similar cases from appearing again

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33123). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Log additional test metrics with the CometCallback
Author: Lothiraldan
Pull request: yes

# What does this PR do?

This PR updates the CometCallback to log metrics when checking performance using the test dataset.
It also updates the CometCallback to use the same metric naming convention as WandbCallback and MLflowCallback.

This is a new PR for https://github.com/huggingface/transformers/pull/32683 after I accidentally deleted our internal fork, sorry about the noise.

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33124). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[WIP] Add support for XTR
Author: shauray8
Pull request: yes
Labels: name: New model

# What does this PR do?

XTR (ConteXtualized Token Retriever) is a multi-vector retrieval model that improves efficiency by focusing on retrieving and ranking the most important document tokens. Details are described at https://arxiv.org/abs/2304.01982.
Author: @jhyuklee

Fixes #31873

## Before submitting
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@ArthurZucker @amyeroberts


Comments:
cc @ArthurZucker 

@shauray8 Thank you for working on the request I opened. Are you still planning to resolve the failing checks? If there's anything I can help with, please feel free to let me know!
Make qwen2 `attention_qkv_bias` optional
Author: wavy-jung
Pull request: yes

# What does this PR do?
Qwen2 and Qwen2-MoE model is forced to add bias to the query, key and value linear projections.
However, following the trend with other recent models (e.g. llama), I refactored these `attention_qkv_bias` to be optional so that we can configure it in config file.

Fixes #32892 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@ArthurZucker @stevhliu 

Comments:
```
RuntimeError: Failed to import transformers.models.audio_spectrogram_transformer.feature_extraction_audio_spectrogram_transformer because of the following error (look up to see its traceback):
```
I have no idea why the check_repo test failed with the error messages above. Can you please give me some guides?
to: @ArthurZucker @stevhliu 

cc @ArthurZucker 

@ArthurZucker Thank you for your response!
What I intended is to allow the option to be set/unset in qwen2 as well. In fact, the LLaMA model has never used a bias in linear projection across all its series, and in the official repository, they are all set to `bias=False`. However, in the Hugging Face implementation, this can be set/unset through the config. Therefore, I want to provide the same functionality here as well.

@ArthurZucker Hey, do you have any further comments for this?

Hey! The reason we added that param for llama is for the release of #26302: InternLM was officially using this flag, meaning there is a model related to this change! 
Unless a model is publish we usually avoid just enabling something that does not have a model associated! 🤗
Implementation of SDPA for Microsoft Phi-3 Mini
Author: Dev4011
Pull request: no
Labels: name: Feature request

name: SDPA

### System Info

- `transformers` version: 4.42.3
- Platform: Linux-6.1.85+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.3
- Accelerate version: 0.32.1
- Accelerate config: not found
- PyTorch version (GPU?): 2.3.0+cu121 (True)
- Tensorflow version (GPU?): 2.15.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.8.4 (gpu)
- Jax version: 0.4.26
- JaxLib version: 0.4.26
- Using distributed or parallel set-up in script?: Yes
- Using GPU in script?: Yes
- GPU type: Tesla T4


### Who can help?

@ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am referring to the below tutorial to finetune Microsoft Phi3 on my custom dataset: https://github.com/microsoft/Phi-3CookBook/blob/main/code/04.Finetuning/Phi-3-finetune-lora-python.ipynb

As I am doing it on Colab on T4 GPU, the Flash Attention is not supported yet [FlashAttention only supports Ampere GPUs or newer.]

Thus, according to the below code from tutorial, attention_implementation is selected as 'sdpa' with compute datatype as torch.float16

if torch.cuda.is_bf16_supported():
  compute_dtype = torch.bfloat16
  attn_implementation = 'flash_attention_2'
else:
  compute_dtype = torch.float16
  attn_implementation = 'sdpa'

# Loading Model
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=compute_dtype, trust_remote_code=True, device_map='auto',
    attn_implementation=attn_implementation
)

# Error
It gives me the error: ValueError: Phi3ForCausalLM does not support an attention implementation through torch.nn.functional.scaled_dot_product_attention yet. Please request the support for this architecture: https://github.com/huggingface/transformers/issues/28005.

and keeping attention_implementation='eager' leads to CUDA Out of Memory error.

### Expected behavior

SDPA should be supported as an Attention Implementation for Microsoft Phi3 model

Comments:
One thing to not is that `Phi3` uses `trust_remote_code=True`, meaning not sure it's compatible with transformers. 
AFAIK it is, se removing the `trust_remote_code=True`, should work

I think this can be changed to `True`?
https://github.com/huggingface/transformers/blob/3d7c2f9dea45338b7ebcd459b452e2fad7abfa1f/src/transformers/models/phi3/modeling_phi3.py#L844
I confirmed that model can be loaded properly after changing that to `True`.

Yep, can you open a PR? 🤗 

@ArthurZucker Opened a PR!

Hi, I changed the _supports_sdpa to True, but I still get this error. 

I am using V100. transformers version 4.42.4. How should I change the code? Thanks

Oh, I set the trust_remote_code=False and set the _supports_sdpa to True. It works now.

Thanks for updating!
[docs] add quick usage snippet to Whisper.
Author: Vaibhavs10
Pull request: yes

# What does this PR do?
The aim of this PR is to add a simple code snippet that anyone stumbling upon the Whisper documentation can look at and get to "Whispering" ASAP.

Internal [slack thread](https://huggingface.slack.com/archives/C04F8N7FQNL/p1717596323781449) for context

Once we agree on a structure I'd like to open a more bigger PR to add such for top visited text-generation models and image/ VLMs too.

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community can review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@LysandreJik @stevhliu @osanseviero @pcuenca 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31289). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Not stale

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Not stale
Bug in transformers v4.43.3: 'p_mask' variable type mismatch in pipelines/question_answering.py
Author: 8libra
Pull request: no
Labels: name: Good First Issue

name: bug

### System Info

Transformers v4.43.3, Python 3.12.4, Debian Linux

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1.  Install transformers v4.43.x.

2.  Trigger the bug with this script:

```python
#! /usr/bin/env python
from transformers import pipeline
checkpoint = "impira/layoutlm-document-qa"
nlp = pipeline("document-question-answering", model = checkpoint)

url = "https://templates.invoicehome.com/invoice-template-us-neat-750px.png"
question = "What is the invoice number?"
p = nlp(url, question)
```

3.  The issue is found when Comparing .../site-packages/transformers/pipelines/question_answering.py between transformers v4.42.4 and transformers v4.43.3. Here's the diff:

```diff
121c121
<     undesired_tokens = np.abs(np.array(p_mask) - 1)
---
>     undesired_tokens = np.abs(p_mask.numpy() - 1)
```

The variable `p_mask` is of type `list` and has no numpy() method.


### Expected behavior

No crash

Comments:
Hey! would you like to open a PR to fix this? 

Hi @ArthurZucker , I have checked the code. It seems this issue has been fixed in the current version of code.
Please refer to [this line of code](https://github.com/huggingface/transformers/blob/3bf6dd8aa141ff6bebd579649e9d0ba8bf5d03db/src/transformers/pipelines/question_answering.py#L121)

Thanks. I’m going to mark this as close.
Update stateful_callbacks state before saving checkpoint
Author: brs-pt
Pull request: yes
Labels: name: trainer

### Fixes
This PR addresses an issue where stateful callbacks, such as EarlyStoppingCallback, were not being updated before saving checkpoints. As a result, resuming training would not have access to the latest state of these callbacks.

### Description
Updated the state of stateful callbacks before saving checkpoints to ensure that their state is preserved and correctly restored when resuming training.

### Suggested Reviewers
@muellerz  @amyeroberts  @SunMarc


Comments:
Gentle ping @muellerzr 

@pedrobrs can you rebase from main? This should fix the failing tests

looks like this logic broke some of the tests in test_trainer.py:
```
 tests/trainer/test_trainer_callback.py::TrainerCallbackTest::test_stateful_duplicate_callbacks
```

Looks like this is because this change doesn't actually wind up restoring the state?


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_32115). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I saw that error when pushing the first time, I'll investigate the issue to identify and fix the problem.
There is at least one other test failed t(marked as required)  that I don't fully understand. The error occurs in step_and_quality / check_repository_consistency. The error message is:

RuntimeError: Failed to import transformers.models.audio_spectrogram_transformer.feature_extraction_audio_spectrogram_transformer because of the following error (look up to see its traceback):
libtorch_cuda.so: cannot open shared object file: No such file or directory

I may not be able to resolve this issue on my own. What should I do about it?



This one is unrelated to you and was already fixed on main!

cc @ArthurZucker for final 🚀
Spelling Error in clean_up_tokenization_spaces was not set
Author: Xemorr
Pull request: no
Labels: name: bug

### System Info

N/A

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

Do not include clean_up_tokenization_spaces parameter in Tokenizer

### Expected behavior

Error is produced, but the error is spelt correctly. 
The word deprecated should be used instead of depracted


Comments:
Thanks for raising! This has been resolved on main #32484

Should this be closed then now that the PR is merged?

Closing as merged yes!
llama3 position_ids error with left padding
Author: ummagumm-a
Pull request: no
Labels: name: Feature request

### Feature request

The LLaMA 3 implementation should generate default `position_ids` that take the `attention_mask` into account.

@ArthurZucker @younesbelkada 

### Motivation

Is there a specific reason why the default `position_ids` generation doesn’t consider the `attention_mask`? A friend mentioned that this issue has persisted for almost half a year now.

https://github.com/huggingface/transformers/blob/adb91179b9e867b7278e0130c87558974056c7b4/src/transformers/models/llama/modeling_llama.py#L962

The problem arises when using left padding, as the default position_ids start from the first index in the sequence length  rather than from the first non-zero index in the `attention_mask`.

As far as I know, this is handled correctly in the `generate` function, but I would expect consistency during training as well.
https://discuss.huggingface.co/t/llama-position-ids/75870

### Your contribution

I can submit a PR changing 

https://github.com/huggingface/transformers/blob/adb91179b9e867b7278e0130c87558974056c7b4/src/transformers/models/llama/modeling_llama.py#L963

to

```
position_ids = (attn_mask.cumsum(-1) - 1).clamp(min=0)
position_ids.masked_fill_(attn_mask.to(torch.bool) == 0, 0)
```

if noone has any objections on correctness of that. Otherwise, let's discuss.

Comments:
+1 
same problem, need solution 

Hey! feel free to open a PR, this should only be activated when training. I am not certain, but we try to avoid handling too many things as it bloats the codebase. Also in general when you train you have a lot more control on what you want to achieve with the position ids. 

It's the first time I have seen a report about that TBH 🤗
Set position_embeddings in BertEmbeddings for absolute position type only, to avoid unused parameters
Author: noam-weinberger-liftoff
Pull request: yes

The position_embeddings are only used for the "absolute" position type. For other position types, they are unused, causing DDP training to fail due to unused parameters. Instead, we should only add this module if it will be used in the forward pass.

Tagging @ArthurZucker, because this is a text model, but feel free to tag others instead.

This is my first contribution, so please let me know if I've missed a step (or worse)!

Comments:
@ArthurZucker Thanks for the response. I took a stab at adding the position_embeddings to the `keys_to_ignore_on_load_unexpected`. However, I'm really not familiar with how this field is used, to this was mostly a guess.
Chameleon image generation low quality. 
Author: Mr-Loevan
Pull request: no
Labels: name: Feature request

name: WIP

name: bug

### System Info

transformers 4.43.0 dev

### Who can help?

@zucchini-nlp 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Using the model.generate function within the transformers framework to generate an image, in comparison to the inference pipeline from https://github.com/GAIR-NLP/anole, results in a relatively lower quality image.

Besides I take a test on some image generation benchmark, and get consistent results.

### Expected behavior

1. Could you please provide a script to convert Hugging Face weights to PyTorch weights, so that we can use them with the Meta inference pipeline?
2. Could you kindly outline the best sampling parameters in the doc for generating high-quality images?

Comments:
Hey @Mr-Loevan , 

Chameleon currently doesn't support image generation as it wasn't releaseed by Meta. As you mentioned, Anole has tuned the model for image generation, but it is still in progress and will be added to the library soon. You can track the progress here (https://github.com/huggingface/transformers/pull/32013)

Regarding the conversion of weights, we don't have a script to convert HF-format to Meta-format. I am not sure why you would need that tbh, afaik meta pipeline neither supports image generation