noamaanMulla-03/transformers-issues huggingface dataset:
fix cpu bnb path
Author: jiqing-feng
Pull request: yes

Hi @Rocketknight1 @ArthurZucker @SunMarc @gante 

We have "cpu" in hf_device_map when using bnb model in CPU. The cpu device bnb model should be accepted by transformers because CPU backend has been enabled in BNB. Please take a review, thx!

Comments:
Looks good, but do you have an example of code that failed before, that is fixed by this PR?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34647). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> Looks good, but do you have an example of code that failed before, that is fixed by this PR?

I run this case in a CPU-only device
```python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig

model_id = "Felladrin/Llama-68M-Chat-v1"

text = ["I am happy because", "This is"]
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token
input_ids = tokenizer(text, return_tensors="pt", padding=True)

quantization_config = BitsAndBytesConfig(load_in_8bit=True)

model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", quantization_config=quantization_config)
model.generation_config.cache_implementation = "static"
model.generate(**input_ids)
```

Error: `IndexError: list index out of range`

Hi @Rocketknight1 . The script can reproduce this error easily both on AWQ and BNB. Besides, it's not safe to get index without checking the list length.

Hi @Rocketknight1 @ArthurZucker @SunMarc @gante @zucchini-nlp , do you mind reviewing this change? Thanks

Hi @Titus-von-Koeller . This change is needed for bitsandbytes cpu path, can you help to review it?

Besides, it's also needed for AWQ cpu path which is already enabled here: #33460 

The following script can reproduce the AWQ error:
I run this case on a CPU-only device.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig, AwqConfig

model_id = "PrunaAI/JackFram-llama-68m-AWQ-4bit-smashed"

text = ["I am happy because", "This is"]
tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token
input_ids = tokenizer(text, return_tensors="pt", padding=True)

# quantization_config = BitsAndBytesConfig(load_in_8bit=True)
quantization_config = AwqConfig(version="ipex")

model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto", quantization_config=quantization_config)
model.generation_config.cache_implementation = "static"
model.generate(**input_ids)

```

Hi @aymeric-roucher  @LysandreJik , do you have time to review this bug fix? Thanks!

@jiqing-feng sorry for late reply, the `transformers` team was off last week and someone will review the PR soon. Seems like you tagged all relevant people already

Hi @SunMarc , I have applied your changes, thanks!
change bnb tests
Author: jiqing-feng
Pull request: yes
Labels: name: Quantization

1. BNB for CPU and XPU path do not support autocast lora finetune for now.
2. XPU do not support gpt2 for now.
3. Add llama tests

Comments:
Thanks for this great work in conjunction with [BNB PR #1418](https://github.com/bitsandbytes-foundation/bitsandbytes/pull/1418), @jiqing-feng 🔥🤗

I'll do my best to provide feedback on this ASAP so we can iterate. That said, I need to balance it with other high-impact topics like quantization improvements, the custom_ops registration refactor (which underpins merging all this into `main` on BNB) and general maintenance (e.g., resolving the currently broken CI integration tests). Still, this remains one of our top priorities before the end of the year, and we’re aiming to make maximum progress on this topic and bring these new functionalities live ASAP.

Thanks so much to you and the Intel team ❤️ for your continued valuable work and support on this! It’s highly appreciated, and I’m looking forward to a final sprint to materialize the fruits of our collaboration.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34713). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Will review this today and tmr.

I'm going to take over for reviewing this. Will work on getting access to hw to run this on.
 cc: @SunMarc 

Hi @matthewdouglas . Thanks for your testing.
When you said before this PR, does it means you observed the failed issue without this PR and the failed issue disappear with this PR?
Do I need any changes before merging?

@jiqing-feng That's correct: I observed that with this PR applied the tests run correctly.. No changes needed. Thanks!
Enables CPU AWQ model with IPEX version.
Author: jiqing-feng
Pull request: yes

This PR enables CPU AWQ model with IPEX version.

Comments:
Hi @SunMarc . Do you mind reviewing this PR? It enables AutoAWQ CPU path. Thx!

Hi @SunMarc @ArthurZucker . This PR is ready to be reviewed. Thx!

Hi @SunMarc . I have fixed all your comments, please review it. Thx!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_33460). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @SunMarc . I have fixed the log, now it's ready to merge. Thanks!
update doc
Author: jiqing-feng
Pull request: yes

Update CPU doc

Comments:
cc @stevhliu 

Hi @stevhliu . Thanks for your review. I have fixed all the comments, now it's ready to merge :)

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34478). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
enable QA bf16 pipeline
Author: jiqing-feng
Pull request: yes

Hi @SunMarc @ArthurZucker 

The same issue as #31625 . Numpy can only process fp32 data type but the input tensor could be bf16/fp16 based on the model's dtype. Please review it, thx!


Comments:
The issue can be reproduced by:
```python
import torch
from transformers import pipeline

question = "How many programming languages does BLOOM support?"
context = "BLOOM has 176 billion parameters and can generate text in 46 languages natural languages and 13 programming languages."

question_answerer = pipeline("question-answering", model="bert-large-uncased-whole-word-masking-finetuned-squad", torch_dtype=torch.bfloat16)
out = question_answerer(question=question, context=context)
```

Hi @Rocketknight1 , could you please review this PR ? Thx!


Merging without core maintainer review since this is a straightforward pipeline dtype fix that shouldn't have any side effects

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34483). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
enable low-precision pipeline
Author: jiqing-feng
Pull request: yes

Hi @amyeroberts @Narsil .

As the previous PR #31444 mentioned, it could enable low-precision pipelines by converting the outputs to `float()`. I followed the codes in [here](https://github.com/huggingface/transformers/blame/main/src/transformers/pipelines/text_classification.py#L205-L210). Do you mind taking a review? Thx!


Comments:
Related to https://github.com/huggingface/transformers/pull/31342 but I dont quite get your changes - what exactly does it fix? When I tested all the pipelines in fp16 none of them had issues outputting logits

> Related to #31342 but I dont quite get your changes - what exactly does it fix? When I tested all the pipelines in fp16 none of them had issues outputting logits

Hi @aliencaocao , FP16 works fine but BF16 is not acceptable for numpy, you can see:
![image](https://github.com/huggingface/transformers/assets/107918818/f5e3e98e-cddd-4736-9017-fcf78fe130d5)




Hi @SunMarc. I have fixed all your comments; please review them. BTW, the failed tests are due to the connection error, not related to my changes.

Hi @amyeroberts , would you please review this PR? Thx!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31625). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Hi @amyeroberts . I have passed all tests, do you mind take a review and merge it? Thx!

Hi @amyeroberts . This PR should be ready to merge, please take a review, thx!

Hi @jiqing-feng, thanks for opening this PR! I'll get to reviewing it soon, but will likely be in a few days.

Hi @amyeroberts . Do you mind reviewing this PR? Thx.

Hi @amyeroberts . Do you think it could be merged?

@amyeroberts , could you help review this PR? Thx.

@SunMarc , do you have a suggestion any other people from HF can help review and merge this PR? Seems @amyeroberts has no bandwidth on this these serval months. Thx.

Sorry for the delay @yao-matrix!

@Rocketknight1 is managing the pipeline; Matt, would you mind reviewing this PR when you have a second?

> Overall this PR seems good to me! However, I prefer `if outputs.dtype in (torch.bfloat16, torch.float16)` rather than `if outputs.dtype == torch.bfloat16`, so we can catch `float16` as well.
> 
> Other than that, I'm happy with it!

Hi @Rocketknight1 , thanks for your review, I have fixed it by your comments.

Hi @Rocketknight1 , do you mind helping to re-run the tests? Thx.

Hi @jiqing-feng, another PR at #33554 touched the same files. I'm sorry - I didn't realize that it was doing the same thing as this PR! That PR has been merged, so I've merged this PR with the code there to avoid conflicts.

@jiqing-feng tests pass now and this looks good - are you okay for us to merge it?

Also cc @LysandreJik for final review, but no rush!

> @jiqing-feng tests pass now and this looks good - are you okay for us to merge it?
> 

Yes, please.
check if None before use
Author: jiqing-feng
Pull request: yes
Labels: name: Cache

Hi @gante . We should check if the variable is None before we use it :) 

Comments:
Hi @amyeroberts @gante . Would you please review this PR? Thx!  The failed tests are not related to my changes.

Hi @amyeroberts , would you please take a look and review it? Thx!

> Thanks for fixing @jiqing-feng!
> 
> Could you provide some more context in the PR description on this issue this resolves, ideally with a small code snippet or description of the scenario?
> 
> Could you also add a test which would have caught this?

I see you have fixed it by checking `if model_kwargs.get("past_key_values") is not None`, which will ensure the cache is not None; it might be fixed in some updates when I rebase this PR since this PR has opened for almost a month. I am closing this PR since it has been fixed. Thx!
Enable ProtST on ESM
Author: jiqing-feng
Pull request: yes

This PR enables [ProtST](https://arxiv.org/pdf/2301.12040.pdf) on ESM

Comments:
cc @Rocketknight1 

Hi @jiqing-feng! The ProtST paper looks good but I don't think we should add classes to ESM like this - it will make the ESM codebase very messy. Instead, why not add ProtST as a custom model, and just copy the classes you need from ESM? There are instructions here: https://huggingface.co/docs/transformers/custom_models

Hi @Rocketknight1 , we would like to discuss some more details with you to see if there could be any second thought on ProtST integration to HF. ProtST is a multi-modality model based on ESM and PubMedBERT for protein science. 

As far as we understand, HF does not support multi-modal interface explicitly and directly. People interested in multi-modality learning using LMs can only clone a repo and merge codes from different models by themselves. The main difficulty we believe is separate use of tokenizers for different modality and a lack of an integrating model interface as a general entry for different modality data. 

However, multi-modality has already become the main stream for almost all domains and it may be an interesting idea to have people pile up these existing single-modal models from HF with an easy-to-use interface. 

The public released repo for ProtST is based on TorchDrug, which we strongly agree that is not suitable to be integrated. We have modified the codebase to be under HF. It's still a private undergoing repo, but **we would really like to share with you about it if you feel interested**. 

Overall, we believe supporting multi-modality is very important for HF's development. It's OK no matter whether ProtST would be viewed as appropriate or not to be integrated into the main repo of HF due to its specific application. But we would really like to **push forward the multi-modality part in HF to make the community better**:)



Hi @KatarinaYuan, yes, we agree! The model looks really good and we'd love to have it in the Hugging Face Hub, but there is no need to modify the ESM class code. You can just upload the model as a checkpoint with custom code, without needing to change anything in the `transformers` codebase! Several other important Bio-ML models on the Hugging Face Hub already do this, such as [Nucleotide Transformer V2](https://huggingface.co/InstaDeepAI/nucleotide-transformer-v2-50m-multi-species).

We agree that multi-modal models are important, and we'd be happy to help by answering any questions during the port, or by promoting the model on Twitter after it's added to the Hub.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Assitant model may on a different device
Author: jiqing-feng
Pull request: yes

Hi @gante . Would you please have a look at this PR. The motivation is that I try to put assistant model and self model in different cuda device, or put assistant model on CPU. This PR should enable assistant model on a different device.

I have tested it on both decoder-only model and encoder-decoder model. Could you please help me to review it? Thanks!


Comments:
Hi @gante @amyeroberts . Would you please help me to review it? Thx!
How to use a custom Image Processor?
Author: glamourzc
Pull request: no

I want to use the processor in the form of `auto_map` but when using `AutoProcessor.from_pretrained`, I am unable to load the custom `ImageProcessor`.

The root cause lies in the use of the `transformers_module` to initialize the class in `ProcessorMixin`. 

https://github.com/huggingface/transformers/blob/c7e48053aab09ad11efa2ad12513e9ab56f29563/src/transformers/processing_utils.py#L1018

Even though I have overridden the _get_arguments_from_pretrained method, this issue still exists in the `__init__`. 

https://github.com/huggingface/transformers/blob/c7e48053aab09ad11efa2ad12513e9ab56f29563/src/transformers/processing_utils.py#L383

Perhaps I could avoid inheriting from ProcessorMixin, but I would like to know if there is a more elegant way to achieve this functionality?

Comments:
Slightly may be related to https://github.com/huggingface/transformers/issues/34307#issuecomment-2547746188. There is a way to do that by adding `transformers.MyImageProcessor = MyImageProcessor` in the `processor_mymodel.py` file, but apparently it causes errors in certain import order
AutoModelForDepthEstimation/DepthAnythingDepthEstimationHead unexpected behavior in JIT
Author: sarmientoF
Pull request: no
Labels: name: bug

name: Vision

### System Info

macos/linux cpu/nvidia gpu

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```python
from transformers import AutoImageProcessor, AutoModelForDepthEstimation

image_processor = AutoImageProcessor.from_pretrained("depth-anything/Depth-Anything-V2-Small-hf")
model = AutoModelForDepthEstimation.from_pretrained("depth-anything/Depth-Anything-V2-Small-hf")
model.config.return_dict = False

traced_model = torch.jit.optimize_for_inference(
    torch.jit.trace(model, (torch.rand(1, 3, 518 * 1, 518 * 1)), strict=False)
)
traced_model(torch.rand(1, 3, 518 * 2, 518 * 2))[0].shape # output: (1, 3, 518 * 1, 518 * 1)
````

### Expected behavior

```python
traced_model(torch.rand(1, 3, 518 * 2, 518 * 2))[0].shape # output: (1, 3, 518 * 2, 518 * 2)
```

Comments:
Hi @sarmientoF, thanks for submitting the issue!

Are the shapes different for non-tranced models? 
DepthAnything output depth map is downsampled and needs upsampling to be the same spacial shape as the image.

see code snipped [here](https://huggingface.co/depth-anything/Depth-Anything-V2-Small-hf)
```python
...
# interpolate to original size
prediction = torch.nn.functional.interpolate(
    predicted_depth.unsqueeze(1),
    size=image.size[::-1],
    mode="bicubic",
    align_corners=False,
)
```

It also can be handled by image processor's method `post_process_depth_estimation` ([code](https://github.com/huggingface/transformers/blob/3ea3ab62d80d91f9bdd16bd3cacd8133fb0d4566/src/transformers/models/dpt/image_processing_dpt.py#L475C9-L514))


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix documentation for ColPali
Author: tonywu71
Pull request: yes

# What does this PR do?

This PR fixes:
- a typo in the ColPali quickstart in the model card
- clean the documentation for ColPali.

## Who can review?

- @yonigozlan 
- @ArthurZucker

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35321). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update comment CI bot
Author: ydshieh
Pull request: yes

# What does this PR do?

Let it run the first job only if:

 - the PR is still in `open` state
 - the comment body contains `run-slow` (or its variants)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35323). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
PaliGemma: Make sure to add <eos> to suffix if <image> is present in `text`
Author: probicheaux
Pull request: yes

# What does this PR do?

Solves: https://github.com/huggingface/transformers/issues/35200

Makes sure to add the `<eos>` token to the `suffix` in either case (`<image>` is present in `text` or not). This avoids gotchas when finetuning.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ArthurZucker @molbap 

Comments:
Tested locally with reproducer from https://github.com/huggingface/transformers/issues/35200 and it fixes the issue

Let me know if there's anything else I can do to help get this merged
Improve batching speed for Whisper models using VAD based chunking
Author: Jiltseb
Pull request: no
Labels: name: Feature request

### Feature request

This feature request aims to improve the speed of Whisper's batched version by adding a VAD model (such as pyannote or from NeMO or Silero) and merging chunks up to 30 sec, instead of relying on the Sliding window technique that takes up more time. 

### Motivation

In addition to the benefits stated above, the semantic chunks take care of the transcription at the 30 sec boundaries. 

### Your contribution

A similar batching has already been implemented in [faster-whisper project](https://github.com/SYSTRAN/faster-whisper/blob/c2a1da1bd94e002c38487c91c2f6b50a048000cf/faster_whisper/transcribe.py#L361) with Silero VAD. The WER and Speed on internal test data and [youtube-commons-asr-eval](https://huggingface.co/datasets/mobiuslabsgmbh/youtube-commons-asr-eval) (for long-form transcription) are better on this implementation than the current HF implementation.

Comments:
@ylacombe and @eustlb

@eustlb any update on this front?
Hi
Author: nosu3380
Pull request: no

Hi 
   I have installed the TGI based on Nivida RTX4090, if hope to use text-generation-launcher to trigger the op which is written by triton(such as server/text_generation_server/layers/attention/flash_attn_triton.py), pls tell me the whole command, thanks .

_Originally posted by @alanguo1234 in https://github.com/huggingface/text-generation-inference/discussions/2759_

Comments:
Fix some fa2 tests
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Since the FA2 interface changed, it's not useful to test the classes anymore

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35340). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[`Mamba2`] Varlen implementation
Author: vasqu
Pull request: no
Labels: name: Feature request

### Feature request

Use varlen implementations (cu_seq_lens) of mamba2 and conv1d when requirements are met, i.e. mostly version dependencies.

### Motivation

It's similar to how fa2 works with varlen and it should boost performance while guaranteeing correct performance on batches.

### Your contribution

I can make a PR, not sure when I'll get to it - prolly after xmas days.

Comments:
cc @molbap
Add Dinov2 with registers
Author: BernardZach
Pull request: yes
Labels: name: New model

name: Vision

name: run-slow

# What does this PR do?

This PR is a continuance of #32905 by @NielsRogge. 

When running pytest there were two errors:

ERROR examples/research_projects/codeparrot/scripts/tests/test_deduplicate.py
ERROR templates/adding_a_missing_tokenization_test/cookiecutter-template-{{cookiecutter.modelname}}/test_tokenization_{{cookiecutter.lowercase_modelname}}.py

I am not sure what the problem for these are. Any guidance would be appreciated.

**Relevant Reviewers**
@ArthurZucker 

Comments:
Hi @BernardZach could you add me as a collaborator to your branch? Will push some fixes

@NielsRogge I believe I added you in the correct spot. Let me know if there are any issues.

Hi @BernardZach I've investigated the issue, when running the modular converter, it changes the modeling code in a non-equivalent way, causing various tests to fail. I've pinged @Cyrilvallez about it

Hi all, I was wondering if there are any updates on this PR?

Fixed all tests :) Hmm for some reason my push made this PR closed. My branch is here: https://github.com/huggingface/transformers/compare/main...NielsRogge:add_dinov_2_registers_innovationcore?expand=1.

I wanted to push it here by doing `git push innovationcore main` but for some reason that closed your PR. The issue might be that you're working on the main branch - normally a PR is never on the main branch


I've set you as remote, git remote -v gives:
```
innovationcore  git@github.com:innovationcore/transformers.git (fetch)
innovationcore  git@github.com:innovationcore/transformers.git (push)
```

Opened #35348