noamaanMulla-03/transformers-issues huggingface dataset:
Using SparseAdam with LLaMA
Author: Aradhye2002
Pull request: no

Hi,

I'm working on PEFT techniques. I was trying out a line of work to sparsify gradients during backpropagation to reduce memory consumption. The model I'm using is llama. However I'm facing many issues:

1. SparseAdam, the optimizer for dense parameters but sparse gradients (my usecase), is not supported by the transformers.Trainer class. However I worked around this by tinkering with the source code and adding an option for SparseAdam.
2. nn.Embedding by default produces dense gradients, however for my usecase I require sparse gradients. Hence I need to set the sparse argument to True. I can't seem to find a high level argument to do this so I went ahead and changed the default value in my local pytorch nn.Embedding to True. This has some obvious issues.
3. But most importantly, even after 1 and 2, there seems to be something missing. It says in the error that "SparseAdam does not support dense gradients, please consider Adam instead" clearly implying that the gradients are still dense.

Can someone help me figure out 3? Solving 1 and 2 would be a bonus.
Thanks.

Comments:
cc @younesbelkada @pacman100 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
AutoModelForCausalLM error with accelerate and bitsandbytes
Author: Altrastorique
Pull request: no

### System Info

Use google colab but connected localy with my computer using jupyter. I have Windows 10,  RTX 3070 and no cuda or cudnn because I didn't succed to make it works :( 

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

!pip install transformers trl accelerate torch bitsandbytes peft datasets -qU
!pip install flash-attn --no-build-isolation

from datasets import load_dataset

instruct_tune_dataset = load_dataset("mosaicml/instruct-v3")

...

model_id = "mistralai/Mixtral-8x7B-v0.1"

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
import torch

nf4_config = BitsAndBytesConfig(
   load_in_4bit=True,
   bnb_4bit_quant_type="nf4",
   bnb_4bit_use_double_quant=True,
   bnb_4bit_compute_dtype=torch.bfloat16
)

model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map='auto',
    quantization_config=nf4_config,
    use_cache=False,
    attn_implementation="flash_attention_2"

)


Error message:
ImportError: Using `bitsandbytes` 8-bit quantization requires Accelerate: `pip install accelerate` and the latest version of bitsandbytes: `pip install -i https://pypi.org/simple/ bitsandbytes`

### Expected behavior

This code works when using the google colab free T4 but don't when I run the code localy. Even if I have installed bitsandbytes and accelerate.

Please help me :)
Thank you very much!

Comments:
Hey 🤗  thanks for opening an issue! We try to keep the github issues for bugs/feature requests.
Could you ask your question on the [forum](https://discuss.huggingface.co/) instead? I'm sure the community will be of help!

Thanks!

I ran into the same issue, and the problem likely is that `torch.cuda` is not available, cf.

[src/transformers/utils/import_utils.py#L739-L747](https://github.com/huggingface/transformers/blob/416fdbad7a79e6b6875fa9384a9f4408ef179742/src/transformers/utils/import_utils.py#L739-L747)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
feat: Upgrade Weights & Biases callback
Author: parambharat
Pull request: yes

# What does this PR do?

- Continuation of old PR #29125

This PR adds a few new functionalities to the Weights & Biases Callback

- Logs Peft and Lora Config to wandb if present
- Adds model parameter counts to wandb config and artifact metadata
- Adds on_predict methods to log prediction metrics
- Prints the model architecture to a file alongside the wandb artifact
- Logs initial and final models to the wandb artifact to full reproducibility
- Adds steps and epoch aliases to checkpoint artifacts

- Here's a [link](https://wandb.ai/parambharat/test-transformers/artifacts/model/model-rg4pcjcv/v3) to the what the logged artifacts look like
- Here's a run [overview page](https://wandb.ai/parambharat/test-transformers/runs/rg4pcjcv/overview?workspace=user-parambharat) with added config and metadata for the run with peft configs logged


## Before submitting
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?

## Who can review?

- @muellerzr
- @pacman100
- @amyeroberts 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30135). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@amyeroberts: I think I was finally able to get all ci checks to pass. The issues were related to type annotations and lazy imports. I have fixed these. Can you please take a look ?

Hey @amyeroberts  , do you know if this PR is going into the next release?

@morganmcg1 Yes, all commits merged into main will be part of the next minor release - v4.41. If there are patch releases before this e.g. 4.40.2, then it won't be part of that release. 

If you want to use this immediately, you can install from source, getting all the most recent commits from main: `pip install git+https://github.com/huggingface/transformers`

This PR causes an error when a custom model with no `save_pretrained()` method uses `Trainer` class.

@jonghwanhyeon Could you open an issue, detailing the errors and linking to this PR for reference? This way we'll be able to properly track if/when it's resolved 

@amyeroberts @parambharat It seems that this _always_ logs the initial model checkpoint to wandb, with no way of turning it off, and uploads the _entire_ model when doing a full fine-tune. This is not intended, is it? See #30896 and #30897

@ArthurZucker I appreciate you reverting this; my W&B storage was immediately full and now I can't access my W&B logs anymore without being immediately redirected to the billing page :/ 
Freeing up space doesn't immediately give me access again either, it's super frustrating

- Tom Aarasen
[Gemma, Llama] Remove unused attribute
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR removes the `spaces_between_special_tokens` attribute from `GemmaTokenizer` and `LlamaTokenizer` since that attribute doesn't do anything. This is because the `_decode` [method](https://github.com/huggingface/transformers/blob/d9850abd407821fb9f4ad5aabb206adad3610f75/src/transformers/tokenization_utils.py#L991-L997) in `tokenization_utils.py` doesn't make use of this attribute, it only accepts a flag which defaults to `True`.

Alternatively, we could make sure that that attribute actually does something, by checking in the `_decode` method whether the tokenizer has it set. Up for discussion

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30327). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Or always set it to `False` and just deprecate it. So many people are confused by it 

Both work hand in hand, we set it to True for old models

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
trainer: log metrics at the end of evaluate()
Author: mdeff
Pull request: yes

Log metrics at the end of `trainer.evaluate()` so that metrics updated by `on_evaluate()` callbacks and `_memory_tracker.stop_and_update_metrics()` are logged.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@muellerzr and @pacman100 (because trainer)

Comments:
Is that a good idea? I can update the tests if it is.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30219). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Redirect transformers_agents doc to agents
Author: aymeric-roucher
Pull request: yes

# What does this PR do?
Redirect the previous `transformers_agents` doc to `agents`.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31054). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Add split special tokens
Author: itazap
Pull request: yes

# What does this PR do?

Fixes #30685 #28648


## Before submitting
 - [ ] make sure this is saved and used not only as kwargs but also the attribute
 - [ ] added tests
 - [ ] TODO: add docs

Edit:
 - [ ] Moved split_special_token testing to common tests, as it should be supported by both fast and slow models.
 - [ ] Verify fast and slow behaviour is identical
 - [ ] Tested with T5
 - [ ] Note: split_special_tokens is expected in kwargs

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30772). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Follow up: Fix link in dbrx.md
Author: eitanturok
Pull request: yes

To followup on PR #30509, I removed this disclaimer "though this may not be up to date" as this link takes you to an up to date version of our code.

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30514). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Docs / Quantization: refactor quantization documentation
Author: younesbelkada
Pull request: yes

# What does this PR do?

As per title, this PR refactors the quantization documentation to make it clearer, less aggressive to users and simple to understand, mainly about which quantization method to use when - still WIP

cc @SunMarc @stevhliu @Titus-von-Koeller 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30942). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

In this PR you deleted the `quantization.md` file, doing so means that all existing links will redirect to a non-existing file (or to the previous file as long as it's cached).

You should update the following file to ensure that users coming from other places don't get redirected to an empty/outdated file: https://github.com/huggingface/transformers/blob/main/docs/source/en/_redirects.yml

cc @younesbelkada 

We likely want to redirect to the overview.md file

Thanks for the heads up ! Done in https://github.com/huggingface/transformers/pull/31063
Fix quanto tests
Author: SunMarc
Pull request: yes

# What does this PR do ? 
This PR fixes quanto tests. With the latest release of quanto and the addition of new W4A16 kernels from AWQ, we do not get the same output. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31062). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix pad_to_max_length Whisper
Author: ylacombe
Pull request: yes

# What does this PR do?
Fix  #30223!

I've added the `device` argument because one of the method using pad_to_max_length is static !

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30787). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hey @amyeroberts, I'm not totally sure about how to properly test auto device mapping, but I've added two tests, one on a single gpu settings, the other on the multi-gpu settings, does it work for you ?
Both were tested and passed on a DGX !

Thanks @amyeroberts ! I still need your approval to merge!
skip `test_model_parallelism` for 2 model test classes
Author: ydshieh
Pull request: yes

# What does this PR do?

We get many failures (` RuntimeError: CUDA error: an illegal memory access was encountered`) in multi-gpu tests since May 24 for rkwv and deformable_detr. It is caused by `test_model_parallelism` which itself cause many other tests to fail.

https://github.com/huggingface/transformers/actions/runs/9248300038/job/25438485848
https://github.com/huggingface/transformers/actions/runs/9248300038/job/25438482290

Let's skip it to have a better CI state. 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31067). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Looks like it is caused by `ninja` - don't know why it is shown up on May 24 but not before in our docker image.
I will remove it.

oh, it's from `quanto`.
Question about VisionEncoderDecoderModel
Author: tannq-jo
Pull request: no

Hi,
I am customizing the VisionEncoderDecoderModel class in the HuggingFace library. In the __init__ function of this class, at line 189, I saw config.tie_word_embeddings is set to False. After extensive research, I still can't understand why we must not tie the input and output embeddings since the input and output embeddings are from the decoder and the decoder is an XForCausalLM class's object, it will tie the input and output embeddings eventually. Could you please explain it to me? It would be a great help for me.

Thank you for your attention to this matter.

https://github.com/huggingface/transformers/blob/fbd8c51ffcc23611f99f5a75fe232f1b010eeb72/src/transformers/models/vision_encoder_decoder/modeling_vision_encoder_decoder.py#L189

Comments:
Hi, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

cc @ydshieh who I believe added this model originally

> since the input and output embeddings are from the decoder and the decoder is an XForCausalLM class's object,

I'm not sure this is completely correct - the input to the model are pixel values, and the vision encoder will have its own embeddings, which hare not related to the decoder's embeddings. 

It turns out that it is not me added this modeling file, it's @NielsRogge 😄 

https://github.com/huggingface/transformers/pull/13874/files#diff-d6b5547bc8ece59fd6c96645e48cce4f32d0995a6621b57f896ad800168017bd

In such composite modeling cases, the encoder and decoder are usually from 2 different models (well, we could use the same model if it is `EncoderDecoderModel`), and the input/output tie embedding doesn't really make sense as @amyeroberts  mentioned.

You can definitely tweak the code to suit your custom modeling logic and needs. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Update feature request label in template
Author: amyeroberts
Pull request: yes

# What does this PR do?

Updates the label name in the feature request issues. 

At the moment, a label wasn't automatically being added to the new issue when create, as `"feature"` is not one of our labels

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_30940). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Remove `ninja` from docker image build
Author: ydshieh
Pull request: yes

# What does this PR do?

See #31067.

In short: `quanto` installs `ninja` which cause some `CUDA error: an illegal memory access` errors. @zucchini-nlp checked that we don't need `ninja` for `QuantizedCache` and its tests.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31080). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
fix from_pretrained in offline mode when model is preloaded in cache
Author: oOraph
Pull request: yes

# What does this PR do?


Fixes # (issue)
fix model load in offline mode

## Who can review?
@ArthurZucker
@Wauplin 

Reproduce:
1. download a private model from hub
2. script:
```
import os

os.environ['TRANSFORMERS_OFFLINE'] = "true"

from transformers.models.auto.modeling_auto import AutoModelForImageClassification

model_id = "Narsil/private"
revision = "main"
kwargs = {'use_auth_token': None}


model = AutoModelForImageClassification.from_pretrained(model_id, revision=revision, **kwargs)
```


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31010). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Thanks! Can you add this example by using HF hub to downlaod snapshot of a .bin file in a temp repo? and test both the flag to go offline and the local_files_only in from pretrained? 🤗

Done @ArthurZucker , added some unit test -> fails before fix, succeeds after
test is ugly but I could not find any cleaner way to do it...
Remove float64 cast for OwlVit and OwlV2 to support MPS device
Author: qubvel
Pull request: yes

# What does this PR do?

Remove `pred_logits` float64 casting, to support "mps" device for OwlVit and OwlV2.
It seems like we can safely remove float64 cast, two lines later there is a cast to float32.

```diff
- pred_logits = pred_logits.to(torch.float64)
pred_logits = torch.where(query_mask == 0, -1e6, pred_logits)
pred_logits = pred_logits.to(torch.float32)
```

The [orignal repo](https://github.com/google-research/scenic/blob/e08103067d2033470e5d072a0f4117a02f6f9a4a/scenic/projects/owl_vit/layers.py#L249) does not contain float64 cast, but probably there was a specific reason to do that. Super safe variant might be 

```diff
- pred_logits = pred_logits.to(torch.float64)
+ pred_logits = pred_logits.to(torch.float32)
pred_logits = torch.where(query_mask == 0, -1e6, pred_logits)
pred_logits = pred_logits.to(torch.float32)
```


Fixes # (issue)
https://github.com/huggingface/transformers/issues/31060

## Who can review?
@LysandreJik @NielsRogge 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31071). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fix OWLv2 post_process_object_detection for multiple images
Author: qubvel
Pull request: yes
Labels: name: run-slow

# What does this PR do?

Fix OWLv2 post_process_object_detection for multiple images. Currently fails due to direct comparison `img_h > img_w`, while that might be tensors with multiple values.

Fixes #31077

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request)
- [x] Did you write any new necessary tests?


## Who can review?

@amyeroberts 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
1. Submitted a test first, that reproduces the error mentioned in the issue
2. Submitted a fix, to pass the test

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31082). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.