noamaanMulla-03/transformers-issues huggingface dataset:
[`RWKV`] Fix RWKV 4bit
Author: younesbelkada
Pull request: yes

# What does this PR do?

Fixes RWKV 4bit inference. 
Fixes https://github.com/huggingface/transformers/issues/23848

```python
from transformers import AutoTokenizer, AutoModelForCausalLM, GenerationConfig

model_id = "RWKV/rwkv-4-1b5-pile"

model = AutoModelForCausalLM.from_pretrained(model_id, load_in_4bit=True, device_map={"":0})
tokenizer = AutoTokenizer.from_pretrained(model_id)

generation_config = GenerationConfig(max_new_tokens=20, pad_token_id=tokenizer.eos_token_id)
question = "Hello my name is Younes"
inputs = tokenizer(question, return_tensors="pt").to(0)
output_int8 = model.generate((inputs["input_ids"]), generation_config=generation_config)
print(tokenizer.decode(output_int8[0], skip_special_tokens=True))
>>> Hello my name is Younes and I am from the United States. I am in the process of applying to the University of California,
```

The fix is similar to 8bit linear layers, one needs to rescale the quantization statistics of the fp4 linear layers

cc @sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
In ViTForMaskedImageModeling, you will receive a reconstructed_pixel_values which shape is different with input when model.config.patch_size is not 16. This further triggers an error about loss when patch_size is not 16 and bool_masked_pos is not None.
Author: yrqUni
Pull request: no

### System Info

- `transformers` version: 4.29.2
- Platform: Linux-3.10.0-1160.90.1.el7.x86_64-x86_64-with-glibc2.35
- Python version: 3.9.16
- Huggingface_hub version: 0.14.1
- Safetensors version: not installed
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Use ViTForMaskedImageModeling;
Set model.config.patch_size except 16;
Get error.

### Expected behavior

In ViTForMaskedImageModeling, you will receive a reconstructed_pixel_values which shape is different with input when model.config.patch_size is not 16. This further triggers an error about loss when patch_size is not 16 and bool_masked_pos is not None.

Comments:
cc @amyeroberts 

Hi @yrqUni, thanks for reporting this issue. 

Digging into this, the error is arising because the decoder head for the model is parametreized by `config.encoder_stride`, which controls the size of the upscaled image. When we update the patch size, in order to calculate the loss, the encoder stride needs to be updated to ensure the reconstructed image has the same resolution as the input. 

I've opened a PR to raise a warning if the loss calculation isn't possible with the configuration settings.
Raise error if loss can't be calculated - ViT MIM 
Author: amyeroberts
Pull request: yes

# What does this PR do?

Currently, `ViTForMaskedImageModeling` will fail when calculating the reconstruction loss if a patch size other than 16 is chosen. This is because the decoder head is parametrized by `config.encoder_stride`, which controls the resolution of the upsampled image. 

By default, `config.patch_size = config.encoder_stride = 16`. If a user updates the patch size but not the encoder stride to match, the reconstructed image will have a different resolution. 

This PR adds a warning for the user before the forward pass, explaining why the loss calculation won't work.

Fixes #23832


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


Comments:
_The documentation is not available anymore as the PR was closed or merged._
Empty circleci config
Author: sgugger
Pull request: yes

# What does this PR do?

This makes sure there is always a circleCI config (and thus always a run_tests workflow) so we can detect it in the branch protection rules.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23913). All of your documentation changes will be reflected on that endpoint.
Bug fix - flip_channel_order for channels first images
Author: amyeroberts
Pull request: yes

# What does this PR do?

The `flip_channel_order` function for rotating the channel order of images from BGR -> RGB had a bug when the input image was in channels first order. 

Previously, the rows of pixels would be rotated, rather than the channel orders i.e. `image = image[:, ::-1, ...]` instead of `image = image[::-1, ...]`.

For the current image processors and pipelines, this path would only be triggered if `do_resize` was overridden and set to `False`. The method is used in 3 model's image processing files: 
* LayoutLMV. If `do_resize=False` no batches could be prepared as the images would be of different sizes and they do no additional cropping or padding.
* LayoutLMV3 (just imported - wasn't used by the image processor)
* MobileViT

This PR:
* Moves the common logic into the image transforms library
* Resolves the bug 
* Adds tests


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Re-enable squad test
Author: sgugger
Pull request: yes

# What does this PR do?

This re-enables the squad test which was skipped while Accelerate was getting fixed. Will merge when all is green.

This also fixes the `[all-test]` directive which was broken by the recent changes for examples in the test fetcher.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23912). All of your documentation changes will be reflected on that endpoint.
[PushToHub] Make it possible to upload folders
Author: NielsRogge
Pull request: yes

# What does this PR do?

For some models, it may be that we have several files with the same name, e.g. for the new InstructBLIP model (#23460), the processor consists of 2 tokenizers (because the model internally uses 2 different text models). Both of these tokenizers require files with the same name, like `tokenizer_config.json`.

Hence, it would be nice to create subfolders in the model repos to store for instance all files of one particular tokenizer (similar to how the Diffusers library does this). For InstructBLIP, I created a separate `qformer_tokenizer` folder for this as can be seen [here](https://huggingface.co/nielsr/instructblip-vicuna-7b/tree/main). I had to adapt the `save_pretrained` and `from_pretrained` methods of `InstructBlipProcessor` to save the files to a separate "qformer_tokenizer" folder, and read them back in. I guess those are very specific to InstructBLIP given that the name of the folder is pretty custom.

However, `push_to_hub` currently doesn't support uploading folders with files. This PR adds this functionality.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Skip device placement for past key values in decoder models
Author: sgugger
Pull request: yes

# What does this PR do?

This PR skips the device placement for the `past_key_values` in big models, which is responsible for a lot of time lost according to the analysis in [this issue](https://github.com/huggingface/accelerate/issues/1394). The idea is that the past key values (one per layer) are all generated on the device of the layer they correspond to and never need to be moved.

Needs to be tested with Accelerate at https://github.com/huggingface/accelerate/pull/1491 (until this PR is merged).

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Accelerate handles both strings and arrays :-) 

thanks boss!
:grey_question: Custom tool creation and pip requirements :grey_question: 
Author: adriens
Pull request: no

# :grey_question: About

i'm currently working on  [Creating a new tool](https://huggingface.co/docs/transformers/en/custom_tools#creating-a-new-tool), ... and this tool will rely on a custom `pypi` package.
In the documentation, you show some classic imports but not custom ones.

# :pray: Question

If I create a custom tool, how to make sure that the final user won't be embarrased by my internal package ?... ie. I would like the end-user to only have to import my custom tool without having to install the `pypi` package, so here comes the (newbee/noob)  question:

> "How to package a custom tool that relies itself on a custom `pypi` package ?"

Thank you in advance for your help.

Comments:
It's not possible, they will need to install the packages required by your tool.

Ok, thanks a lot for your answer @sgugger :pray:
NoBadWordsLogitsProcessor banned_mask and indices not in same device
Author: cauyxy
Pull request: no

### System Info

transformers version: 4.29.2
Platform: Linux-6.1.22-aufs-1-x86_64-with-glibc2.31
Python version: 3.9.2
PyTorch version (GPU?): 2.0.1+cu117 (True)

### Who can help?

_No response_

### Information

- [x] The official example scripts
- [x] My own modified scripts

### Tasks

- [x] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction


import torch
import transformers
from transformers import AutoModelForCausalLM, set_seed
from transformers import GenerationConfig, LogitsProcessorList, NoBadWordsLogitsProcessor


set_seed(0)
torch.set_default_device("cuda")
model_name_or_path = 'facebook/opt-125m'

model = AutoModelForCausalLM.from_pretrained(model_name_or_path, device_map='auto')
tokenizer = transformers.AutoTokenizer.from_pretrained(model_name_or_path, add_prefix_space=True)

human = 'give me a sentence, more than 20 words, containing word pig'
input_ids = tokenizer(human, return_tensors='pt').input_ids
input_ids = input_ids.to('cuda')
bad_words = ['pigs', 'piggy', 'pig', 'piggies', 'piggish']
bad_words_ids = [tokenizer(bad_word, add_special_tokens=False).input_ids for bad_word in bad_words]
processor = LogitsProcessorList([NoBadWordsLogitsProcessor(bad_words_ids=bad_words_ids, eos_token_id=model.config.eos_token_id)])
generation_config = GenerationConfig.from_pretrained(model_name_or_path, max_length=100, do_sample=True)
outputs = model.generate(input_ids, generation_config=generation_config, logits_processor=processor)

output_str = tokenizer.decode(outputs[0],  skip_special_tokens=True)
print(output_str)

### Expected behavior

run success

Comments:
the reason is pytorch default device
```python
import torch

torch.set_default_device("cuda")
print(torch.cuda.is_available())

banned_mask = torch.LongTensor([0, 1, 2])
indices = torch.ones(len(banned_mask))

print(f"banned_mask.device: {banned_mask.device}")
print(f"indices.device: {indices.device}")
```
![image](https://github.com/huggingface/transformers/assets/45619430/389aca59-98c4-4d93-bcaf-434f4cca95b9)
RWKV bug for 8-bit model fine-tuning.
Author: LetianLee
Pull request: no

### System Info

- `transformers` version: 4.30.0.dev0
- Platform: Linux-5.15.107+-x86_64-with-glibc2.31
- Python version: 3.10.11
- Huggingface_hub version: 0.14.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.12.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.9 (gpu)
- Jax version: 0.4.10
- JaxLib version: 0.4.10
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@ArthurZucker  @younesbelkada @sgugger

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

The 8-bit model inference works successfully, but after fine-tuning, the model fails when inferring it again.

Reproduction:
```
import torch
from transformers import AutoTokenizer, RwkvForCausalLM, GenerationConfig
from torch.optim import AdamW

model = RwkvForCausalLM.from_pretrained("RWKV/rwkv-raven-1b5", device_map="auto", torch_dtype=torch.bfloat16, low_cpu_mem_usage=True, load_in_8bit=True) 
tokenizer = AutoTokenizer.from_pretrained("RWKV/rwkv-raven-1b5")

optim = AdamW(model.parameters(), lr=1e-4)

ctx = "Hello my name is Bob"
inputs = tokenizer(ctx, return_tensors="pt").to(0)
model(inputs["input_ids"])   # ok

model.train()
outputs = model(inputs["input_ids"], labels=inputs["input_ids"])
loss = outputs.loss
loss.backward()
optim.step()

model.eval()
model(inputs["input_ids"])   # failed
```
or see my colab code as follows:
https://colab.research.google.com/drive/1l_vNHPd9_Z40dPkhIj5CxgrLhIn1Edyc?usp=sharing

### Expected behavior

After fine-tuning, the model should still work properly.

Comments:
Hi @LetianLee 
Thanks for the issue! 
In fact, you cannot train a model that has been purely loaded in 8bit. In order to apply fine tuning using 8bit / 4bit models, you need to add adapters on top of the model and train these adapters only. 
Please check out the official example of PEFT: https://github.com/huggingface/peft/blob/main/examples/int8_training/Finetune_opt_bnb_peft.ipynb and adapt it to your needs. You may need to manually specify `target_modules=["key", "value", "receptance"]` when defining the `LoraConfig`. Please let us know how it goes

Hi @younesbelkada ,
Thank you for your kind reply and explanation. Since this is the case, I will close this ticket as it is not an issue related to Hugging Face/Transformers. Thank you very much for providing the relevant tutorial. I will now proceed to try the Lora fine-tuning. Thanks!

Thanks so much @LetianLee !
Effectively allow `encoder_outputs` input to be a tuple in pix2struct
Author: fxmarty
Pull request: yes

Be consistent with the type hint https://github.com/huggingface/transformers/blob/fabe17a726bbf6081cfbcc975d8ac451a81f3e2d/src/transformers/models/pix2struct/modeling_pix2struct.py#L1656 (which should rather be `Optional[Union[Tuple[Tuple[torch.FloatTensor]], OrderedDict]]` IMO) and follow what is done with other architectures.

Otherwise, currently, with `return_dict=True`, an error is raised as we later try to access `encoder_outputs.last_hidden_state`: https://github.com/huggingface/transformers/blob/fabe17a726bbf6081cfbcc975d8ac451a81f3e2d/src/transformers/models/pix2struct/modeling_pix2struct.py#L1768-L1781

This is blocking for the ONNX export support.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
[don't merge] Test hfh v0.15.0.rc0
Author: Wauplin
Pull request: yes

Trying to trigger the CI differently. Still stuck on the ci-... branch (see https://github.com/huggingface/transformers/actions/runs/5143134417) so I guess I did something wrong.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_23936). All of your documentation changes will be reflected on that endpoint.
rename DocumentQuestionAnsweringTool parameter input to match docstring
Author: Adam-D-Lewis
Pull request: yes

# What does this PR do?
Rename DocumentQuestionAnsweringTool parameter input to match docstring
<!--
Congratulations! You've made it this far! You're not quite done yet though.


Fixes [# (issue)](https://github.com/huggingface/transformers/issues/23921)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @sgugger

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam

Documentation: @sgugger, @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: @sgugger
- TensorFlow: @Rocketknight1

 -->


Comments:
_The documentation is not available anymore as the PR was closed or merged._
DocumentQuestionAnsweringTool docstring is incorrect
Author: Adam-D-Lewis
Pull request: no

### System Info

N/A

### Who can help?

https://github.com/huggingface/transformers/blob/fabe17a726bbf6081cfbcc975d8ac451a81f3e2d/src/transformers/tools/document_question_answering.py#LL32C9-L34C80 is wrong.  The inputs are image: "Image", question: str.  I was trying transformers agent with a task like 

```
agent.run("Given the pdf at <insert URL>, tell me <insert question>")
```
and it was failing with `TypeError: DocumentQuestionAnsweringTool.encode() got an unexpected keyword argument 'document'` because of the docstring issue.


### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

N/A

### Expected behavior

N/A

Comments:
Thanks for flagging! Would you like to open a PR to fix this? The proper fix should be to rename the input `image` to `document` (not change the description).

Here's a draft https://github.com/huggingface/transformers/pull/23939.  Please correct the rest for me if it's not quite right.  Thanks!

Fixed by https://github.com/huggingface/transformers/pull/23939
Modify device_map behavior when loading a model using from_pretrained
Author: SunMarc
Pull request: yes

# What does this PR do?

Change how `device_map` works in the `from_pretrained` function when we load the model: 
- Training setup: we don't need to pass the `device_map` anymore ('cpu' by default or torch.cuda.local_device() for 4/8 bit model) or we can set `device_map` to the device we want our whole model to be placed on('cpu', 'cuda:0', 0, `torch.device('cpu')`...). 
- Inference setup: pass custom device_map or device_map in `["auto", "balanced", "balanced_low_0", "sequential"]`

Fixes [1412](https://github.com/huggingface/accelerate/issues/1412)

## Who can review?

@sgugger @younesbelkada 


Comments:
_The documentation is not available anymore as the PR was closed or merged._