noamaanMulla-03/transformers-issues huggingface dataset:
Speedup modular conversion w/ multiproc
Author: muellerzr
Pull request: yes

# What does this PR do?

During my last PR, I found that `make fix-copies` was taking a ton of time. Trying to make it a bit faster by using multiproc instead. Let me know what we think.

Times calling `time python utils/check_modular_conversion.py  --fix_and_overwrite`:

* Before (linear): 67s
* After (multiproc): 6s

(Mileage may vary with multiproc, of course)

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @Rocketknight1 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36073). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I had the same thoughts just 15 minutes ago 😅 as far as I understand, files are topologically sorted, so the order of conversion matters, we should probably keep the dependency graph and apply multiprocessing level-by-level

Yep exactly! 


Next level is to code it in rust :)
[tests] further fix "Tester object has no attribute '_testMethodName'" 
Author: faaany
Pull request: yes

## What does this PR do?
Similar to this PR #34910, this PR further removes the unwanted `unittest.TestCase` to fix the following error:

```bash
========================================================== ERRORS ===========================================================
__________________________ ERROR collecting tests/models/vitpose/test_image_processing_vitpose.py ___________________________
../../miniforge3/envs/ipex-ww02/lib/python3.10/unittest/case.py:461: in __hash__
    return hash((type(self), self._testMethodName))
E   AttributeError: 'VitPoseImageProcessingTester' object has no attribute '_testMethodName'
===================================================== warnings summary ======================================================
```

@ydshieh 

Comments:
sure, let me update them in this PR. 

@ydshieh code update. Could you take a look? Thx a lot!

Thank you a lot for this huge clean=up.

I see there are still:
 - llava
 - roberta
 - superglue

having files to change if I don't miss any detail.


You can ignore red CI - I will take care of it.

Hi @faaany 

Just in case :-) Once the 3 places mentioned in

https://github.com/huggingface/transformers/pull/35781#issuecomment-2610020541

being updated from your side, I will take over for CI and merge 🙏 

I will finalize this PR as I think @faaany is off for Lunar New year 🎉 

Only part of files in `tests/models` are changed with very clear purpose of fixing. Merge. Thank you for helping again @faaany and Happy New year~
[docs] fix bugs in the bitsandbytes documentation
Author: faaany
Pull request: yes

## What does this PR do?
When running the example codes on XPU, I got the following 3 errors:

```bash
Traceback (most recent call last):
  File "/home/sdp/fanli/doc_to_fix.py", line 5, in <module>
    model = AutoModelForCausalLM.from_pretrained(model_id, BitsAndBytesConfig(load_in_4bit=True))
  File "/home/sdp/fanli/transformers/src/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
  File "/home/sdp/fanli/transformers/src/transformers/modeling_utils.py", line 4129, in from_pretrained
    model = cls(config, *model_args, **model_kwargs)
TypeError: OPTForCausalLM.__init__() takes 2 positional arguments but 3 were given
```

```bash
Traceback (most recent call last):
  File "/home/sdp/fanli/doc_to_fix.py", line 5, in <module>
    quantization_config = BitsAndBytesConfig(
  File "/home/sdp/fanli/transformers/src/transformers/utils/quantization_config.py", line 431, in __init__
    self.post_init()
  File "/home/sdp/fanli/transformers/src/transformers/utils/quantization_config.py", line 470, in post_init
    raise TypeError("llm_int8_threshold must be a float")
TypeError: llm_int8_threshold must be a float
```

```bash
Traceback (most recent call last):
  File "/home/sdp/fanli/doc_to_fix.py", line 18, in <module>
    model_8bit = AutoModelForCausalLM.from_pretrained(
  File "/home/sdp/fanli/transformers/src/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
  File "/home/sdp/fanli/transformers/src/transformers/modeling_utils.py", line 3633, in from_pretrained
    hf_quantizer.validate_environment(
  File "/home/sdp/fanli/transformers/src/transformers/quantizers/quantizer_bnb_4bit.py", line 103, in validate_environment
    raise ValueError(
ValueError: Some modules are dispatched on the CPU or the disk. Make sure you have enough GPU RAM to fit the quantized model. If you want to dispatch the model on the CPU or the disk while keeping these modules in 32-bit, you need to set `llm_int8_enable_fp32_cpu_offload=True` and pass a custom `device_map` to `from_pretrained`. Check https://huggingface.co/docs/transformers/main/en/main_classes/quantization#offload-between-cpu-and-gpu for more details. 
```

And I also found that the below code returns error that no appropriate model files exit in the repo. Not sure whether we should replace it with `meta-llama/Llama-2-13b-chat-hf`, so I didn't make the change in this PR.
```python
model_double_quant = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-13b", torch_dtype="auto", quantization_config=double_quant_config)
```
If we should update it as well, just let me know. 


cc: @stevhliu


Comments:
@stevhliu Thanks, model checkpoint updated.
[docs] fix outdated example code in `trainer.md`
Author: faaany
Pull request: yes

## What does this PR do?
When going through the code examples in `trainer.md`, I found several bugs. The optimizer related bug should be fixed in #35875.
The rest should be fixed in this PR. :) 

Documentation: @stevhliu


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_36066). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[docs] update not-working model revision
Author: faaany
Pull request: yes

## What does this PR do?
Using the existing example code in doc `Share a Model`, I got the following error: 
```bash
OSError: v2.0.1 is not a valid git identifier (branch name, tag name or commit id) that exists for this model name. Check the model page at 'https://huggingface.co/julien-c/EsperBERTo-small' for available revisions.
```
I think it is not intended to make this example code, so I assume that the revision was deleted after the doc was published. This PR updates the revision using commit hash.  

@stevhliu


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34682). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[docs] add xpu device check 
Author: faaany
Pull request: yes

## What does this PR do?
There are many places in the task guides that only check the cuda device availability. This PR adds xpu besides cuda, but I am not sure whether my current change is the best one, because there are other ways to add xpu as, e.g. 
- add xpu in the inline-comment 
- use accelerate's `get_backend` API to get the underlying device 

Just let me what is your favorite, @stevhliu Once agreed by you, I can submit further commits to replicate the change in other places. 

Comments:
> Thanks for the suggestion!
> 
> I think using `get_backend` may be the easiest option for users, and this will also include other devices such as `mps`.

thanks! code update. Is this okay for you? Then I can change to other places as well.

> Very nice, thank you! 👍

I only made one change: I removed GPU from the comment because GPU is a general term including CUDA, XPU, NPU. And I made the same change at other places as well. Pls have a review, thx a lot! 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34684). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[docs] add XPU besides CUDA, MPS etc.
Author: faaany
Pull request: yes

## What does this PR do?
We should have XPU as one option in device-agnostic. 

Documentation: @stevhliu



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34777). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[docs] make `empty_cache` device-agnostic
Author: faaany
Pull request: yes

## What does this PR do?
This PR uses the device-agnostic implementation from accelerate to empty cache and check device availability. 

Thanks for the review! @stevhliu


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34774). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[docs] use device-agnostic API instead of cuda 
Author: faaany
Pull request: yes

## What does this PR do?
As the title suggests, we could automatically detect the underlying hardware accelerator instead of hard-coding. 

Documentation: @stevhliu



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34913). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[tests] add XPU part to testing
Author: faaany
Pull request: yes

## What does this PR do?
Add XPU environment variable `ZE_AFFINITY_MASK` besides `CUDA_VISIBLE_DEVICES`.

@stevhliu

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34778). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[doc] use full path for run_qa.py 
Author: faaany
Pull request: yes

## What does this PR do?
This PR does 2 things:
- Use full path for `run_qa.py`
- Remove `to_bettertransformers()`

With `to_bettertransformers()`, I will get the following error:
```bash
The class `optimum.bettertransformers.transformation.BetterTransformer` is deprecated and will be removed in a future release.
Traceback (most recent call last):
  File "/home/sdp/fanli/tmp/transformers/test_docs.py", line 14, in <module>
    model = model.to_bettertransformer()
  File "/home/sdp/fanli/tmp/transformers/src/transformers/modeling_utils.py", line 4944, in to_bettertransformer
    return BetterTransformer.transform(self)
  File "/home/sdp/miniforge3/envs/ipex-2.5-new/lib/python3.10/contextlib.py", line 79, in inner
    return func(*args, **kwds)
  File "/home/sdp/miniforge3/envs/ipex-2.5-new/lib/python3.10/site-packages/optimum/bettertransformer/transformation.py", line 229, in transform
    raise ValueError(
ValueError: This model already uses BetterTransformer optimizations from Transformers (torch.nn.functional.scaled_dot_product_attention). As such, there is no need to use `model.to_bettertransformers()` or `BetterTransformer.transform(model)` from the Optimum library. Details: https://huggingface.co/docs/transformers/perf_infer_gpu_one#pytorch-scaled-dot-product-attention.
```
If we should keep `to_bettertransformers()`, just let me know. Thanks! @stevhliu



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34914). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[docs] add explanation to `release_memory()`
Author: faaany
Pull request: yes

## What does this PR do?
As a non-cuda user, it would be good to get explained that `release_memory()` is a device-agnostic implementation.

Documentation: @stevhliu

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_34911). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[docs] use device-agnostic instead of `cuda`
Author: faaany
Pull request: yes

## What does this PR do?
This PR further improves the docs using device-agnostic API instead of cuda. 

Documentation: @stevhliu



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35047). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[docs] use device-agnostic API instead of hard-coded cuda
Author: faaany
Pull request: yes

## What does this PR do?
The same with other PRs. :D 

Documentation: @stevhliu




Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35048). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[docs] fix example code bug
Author: faaany
Pull request: yes

## What does this PR do?
The current code example will give me the following error:

```bash
Traceback (most recent call last):
  File "/home/sdp/fanli/transformers/playground.py", line 110, in <module>
    for value in generator:
  File "/home/sdp/fanli/transformers/playground.py", line 80, in model_inference
    inputs = processor(
  File "/home/sdp/fanli/transformers/src/transformers/models/idefics2/processing_idefics2.py", line 227, in __call__
    raise ValueError(
ValueError: The total number of <image> tokens in the prompts should be the same as the number of images passed. Found 3 <image> tokens and 2 images.
```

This is because the `messages` were used in the previous example, while in this example an additional prompt is added. 


Documentation: @stevhliu


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35054). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[docs] add a comment that offloading requires CUDA GPU
Author: faaany
Pull request: yes

## What does this PR do?
Just as the title suggests. I think it is necessary to mention that offloading requires CUDA again, as users could directly jump to this section.

Documentation: @stevhliu

Comments:
done

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35055). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[tests] make cuda-only tests device-agnostic  
Author: faaany
Pull request: yes

Currently this test is restricted to run on CUDA, but it can also run on other accelerators like XPU. 


Comments:
cc @ydshieh 

LGTM, but is still in draft mode or ready to go?

> LGTM, but is still in draft mode or ready to go?

Yes, we are ready to go. 

@Rocketknight1 Maybe we can merge once you approved?

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_35222). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

This seems significant enough that it might need a core maintainer! cc @LysandreJik @ArthurZucker 

I talked to ArthurZucker offline and we agree that I have some level of permissions to merge PRs like this.
improve efficient training on CPU documentation
Author: faaany
Pull request: yes

## What does this PR do?
This PR improves the CPU efficient training documentation to make it more clear, accurate and up-to-date. Concrete improvements are
- add full names of the CPU instruction sets (e.g. Intel® Advanced Vector Extensions 512 instead of AVX-512)
- add one sentence to explain "mixed precision" 
- add OOB mixed precision training using BF16 and further improvement with IPEX

Comments:
Hi @amyeroberts @stevhliu, could you help review this PR? Thx!

@yao-matrix

@stevhliu thanks so much for the corrections! It looks much better now and I only made one more change as @yao-matrix suggested in the comment. Pls help us to merge this PR. Many thanks!
[`Vilt`] align input and model dtype in the ViltPatchEmbeddings forward pass 
Author: faaany
Pull request: yes

## What does this PR do?
Just like the case in [BlipVisionEmbeddings](https://github.com/huggingface/transformers/blob/main/src/transformers/models/blip/modeling_blip.py#L249), there should also be a dtype alignment in the ViltPatchEmbeddings as well. Otherwise, I would get the `RuntimeError: Input type (float) and bias type (c10::Half) should be the same`, when my "dandelin/vilt-b32-finetuned-vqa" model is loaded in half-precision data type.

## Reproduction
```python
from transformers import ViltProcessor, ViltForQuestionAnswering
import requests
from PIL import Image
import torch 

# prepare image + question
url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)
text = "How many cats are there?"

processor = ViltProcessor.from_pretrained("dandelin/vilt-b32-finetuned-vqa")
model = ViltForQuestionAnswering.from_pretrained("dandelin/vilt-b32-finetuned-vqa", torch_dtype=torch.float16).to("cuda")
# prepare inputs
encoding = processor(image, text, return_tensors="pt").to("cuda")

# forward pass
outputs = model(**encoding)
logits = outputs.logits
idx = logits.argmax(-1).item()
print("Predicted answer:", model.config.id2label[idx])
```


@ArthurZucker and @amyeroberts


Comments:
@yao-matrix

Pls have a review, thx!

Hi @ArthurZucker , could you take a look at this PR? Thx! 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28633). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[tests] enable automatic speech recognition pipeline tests on XPU 
Author: faaany
Pull request: yes

## What does this PR do?
"cuda" and "cuda:0" are hard-coded in `test_whisper_longform` and `test_seamless_v2` of `test_pipelines_automatic_speech_recognition.py`. So when running the unit tests on XPU, I get 2 failing tests:
```bash
================================================= short test summary info ==================================================
SKIPPED [1] tests/pipelines/test_pipelines_automatic_speech_recognition.py:269: test requires TensorFlow
FAILED tests/pipelines/test_pipelines_automatic_speech_recognition.py::AutomaticSpeechRecognitionPipelineTests::test_seamless_v2 - AssertionError: Torch not compiled with CUDA enabled
FAILED tests/pipelines/test_pipelines_automatic_speech_recognition.py::AutomaticSpeechRecognitionPipelineTests::test_whisper_longform - AssertionError: Torch not compiled with CUDA enabled
============================= 2 failed, 40 passed, 1 skipped, 31 warnings in 203.95s (0:03:23) =============================
```

This PR proposes to use the device-agnostic variable `torch_device` instead of the hard-coded "cuda" device. After the fix, 
all unit tests in `test_pipelines_automatic_speech_recognition.py` run through on XPU as shown below: 
```bash
================================================= short test summary info ==================================================
SKIPPED [1] tests/pipelines/test_pipelines_automatic_speech_recognition.py:269: test requires TensorFlow
================================== 42 passed, 1 skipped, 32 warnings in 219.40s (0:03:39) ==================================
```

## Who can review?
@Narsil @ArthurZucker 

Comments:
> LGTM but we might need to add `require_torch_gpu` as it was tested on GPU

If I change `require_torch` to `require_torch_gpu`, this test will be skipped on Intel GPU. But I don't see any reason why this test should only work on NV GPU. Maybe we can change `require_torch` to `require_torch_accelerator` to let CPU save this test, what do you think? 

Alright, I just want to make sure our CI runs it with `cuda` because the values were obtained with it and not 100% sure you always get the same. Though it is not logits some device can get it differently, we'll leave that and I'll merge thanks for the PR

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29308). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
[tests] use the correct `n_gpu` in `TrainerIntegrationTest::test_train_and_eval_dataloaders` for XPU
Author: faaany
Pull request: yes

## What does this PR do?
The below test fails on intel GPU with 8 cards:
```bash
___________________________________ TrainerIntegrationTest.test_train_and_eval_dataloaders ___________________________________

self = <tests.trainer.test_trainer.TrainerIntegrationTest testMethod=test_train_and_eval_dataloaders>

    def test_train_and_eval_dataloaders(self):
        n_gpu = max(1, backend_device_count(torch_device))
        trainer = get_regression_trainer(learning_rate=0.1, per_device_train_batch_size=16)
>       self.assertEqual(trainer.get_train_dataloader().total_batch_size, 16 * n_gpu)
E       AssertionError: 16 != 128

tests/trainer/test_trainer.py:1035: AssertionError
```
This is because the `n_gpu` is 8, but the default `_n_gpus` in `TrainingArguments` is 1 as can be seen [here](https://github.com/huggingface/transformers/blob/main/src/transformers/training_args.py#L1969), leading to a total_batch_size of 16. This test works on CUDA, because CUDA supports DataParallel and would set `_n_gpus` to a value larger than 1 as can be seen [here](https://github.com/huggingface/transformers/blob/main/src/transformers/training_args.py#L1984). But on all other devices except CUDA and CPU, this test will fail. 

To fix this issue, we need to add a conditional check to the test to make this test work on other devices except GPU. Pls have a review: @muellerzr and @pacman100

Comments:
Hi @ArthurZucker , thanks for reviewing my other PRs. Would you like to take care of this one as well? Like other tests, this one is also written with only GPU in mind and fails on other devices like XPU. Thanks a lot!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_29307). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@muellerzr can we proceed with this merge? thanks a lot!