noamaanMulla-03/transformers-issues huggingface dataset:
[`Docs`] Add 4-bit serialization docs
Author: younesbelkada
Pull request: yes

# What does this PR do?

Follow up work from: https://github.com/huggingface/transformers/pull/26037 
Adds few lines in the documentation about serializing 4-bit models on the Hub 

cc @amyeroberts @stevhliu 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28182). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Update `docs/source/en/perf_infer_gpu_one.md`
Author: ydshieh
Pull request: yes

# What does this PR do?

Update `docs/source/en/perf_infer_gpu_one.md` to fix 

> FAILED tests/utils/test_doc_samples.py::TestDocLists::test_sdpa_support_list - ValueError: mixtral should be in listed in the SDPA documentation but is not. Please update the documentation.

Comments:
Add Swinv2 backbone
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR is a continuation of #25799. It adds `Swinv2Backbone`, such that it can be combined with the DPT framework (known as [MiDaS 3.1](https://github.com/isl-org/MiDaS/releases)).

It also required some updates to `modeling_swinv2.py` which are fully backwards compatible (all slow integration tests passing).

To do:

- [x] verify image processor before pushing models to the hub
- [x] transfer checkpoints, add integration test
- [x] investigate how MiDas does forward pass with BEiT backbone and keeping the aspect ratio

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27742). All of your documentation changes will be reflected on that endpoint.
ValueError: Target module WQLinear_GEMM is not supported. Currently, only `torch.nn.Linear` and `Conv1D` are supported.- AWQ Quantisation Issues
Author: Vasanth03
Pull request: no

Hi @casper-hansen 

-> I am trying to train the AWQ quantised model using hugging face trainer.
While using PEFT (LoRA adaptor) the following error pops up.

![Screenshot 2023-12-22 at 12 47 36 PM](https://github.com/huggingface/transformers/assets/59615743/0bf7d637-04a9-4830-8d74-da7fd02b128c)

-> This is the version that I have used !pip install -q -U https://github.com/casper-hansen/AutoAWQ/releases/download/v0.1.6/autoawq-0.1.6+cu118-cp310-cp310-linux_x86_64.whl

Any help is much appreciated. Thanks

_Originally posted by @Vasanth03 in https://github.com/huggingface/transformers/issues/27321#issuecomment-1867330086_
            

Comments:
Hi @Vasanth03, AutoAWQ is not compatible with training yet. It will be in the future though.

Thank you so much @casper-hansen.  And I look forward to its compatibility in the future.
Update YOLOS slow test values
Author: amyeroberts
Pull request: yes

# What does this PR do?

Updates the test values for YOLOS after the merging in of #27663 to resolve failing slow model tests on nightly. 

Some small value changes are expected because of the change of output image size from the image processor. 

As a sense check, plotted the output of the object detection model in the tests to visualise differences to confirm they are small and still sensible: 

 **Old detections**

![yolos_old](https://github.com/huggingface/transformers/assets/22614925/009b8800-356e-4f80-8813-b1b3579abe39)

**New detections**
![yolos_new](https://github.com/huggingface/transformers/assets/22614925/aebf728f-5582-4260-933d-eee2fce87785)



Comments:
Fix the check of models supporting FA/SDPA not run
Author: ydshieh
Pull request: yes

# What does this PR do?

The original check (as test methods) in `tests/utils/test_doc_samples.py` won't run (and didn't run like in #28133) as that file is not impacted by the modeling files (in terms of import relation).

Those 2 checks don't need `torch` at all and could be done in the first stage of check (`check_repository_consistency`)

Comments:
i create the pr(https://github.com/huggingface/transformers/pull/28201) check failed casue by `tests/utils/test_doc_samples.py` faild.
should i merge this commit, repush pr?

@inkinworld Yes, once this PR has been merged into `main` then you can rebase on main or merge in the main branch into yours to include this fix and then push your branch again to trigger a re-run of the CI
Drop `feature_extractor_type` when loading an image processor file
Author: ydshieh
Pull request: yes

# What does this PR do?

`preprocessor_config.json` created in old days like [this](https://huggingface.co/openai/clip-vit-large-patch14/blob/main/preprocessor_config.json) has, for example, `"feature_extractor_type": "CLIPFeatureExtractor",` in it. If that file is for an image processor, during the loading (in `__init__`), it is added as the object's attribute. This is already misleading.

If we save the image processor again, the file will contain `feature_extractor_type` and `image_processor_type`, which is even more confusing. See the example below.

**This PR pop up this attribute during the loading, so it won't be an attribute of the loaded object.**

### To reproduce

```python
from transformers import CLIPImageProcessor
import json


p = CLIPImageProcessor.from_pretrained("openai/clip-vit-large-patch14")
print(getattr(p, "feature_extractor_type", None))
print(getattr(p, "image_processor_type", None))
print("-" * 40)

p.save_pretrained("myclip")
p = CLIPImageProcessor.from_pretrained("myclip")
print(getattr(p, "feature_extractor_type", None))
print(getattr(p, "image_processor_type", None))
```

### Output

**before this PR**
```bash
CLIPFeatureExtractor
None
----------------------------------------
CLIPFeatureExtractor
CLIPImageProcessor
```

**after this PR**
```bash
None
None
----------------------------------------
None
CLIPImageProcessor
```

Comments:
failure is also on `main`.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28195). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.
Fixing visualization code for object detection to support both types of bounding box. 
Author: Anindyadeep
Pull request: yes

# What does this PR do?

In the current documentation of using [transformers for object detection](https://huggingface.co/docs/transformers/tasks/object_detection). There belongs a section where we visualize the bounding boxes. But when the same code is used in a different dataset with un-normalized bounding box, it does not work. For a new comer, or someone trying to see results with different dataset might cause friction. So, this PR tries to cover up that edge case.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #27817


## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [X] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Documentation: @stevhliu and @MKhalusova


Comments:
Thank you for your contribution! Please make sure the CI checks are green: run `make style` to fix the issues. 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27842). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thank you for making the example more robust to different datasets! LGTM
update the logger message with accordant weights_file_name
Author: izyForever
Pull request: yes

# What does this PR do?

Update the logger message with accordant weights_file_name.

Fixes # (issue)
https://github.com/huggingface/transformers/issues/28076

@amyeroberts


Comments:
@amyeroberts 

I can't figure out why I failed the tests_torch for I only changed the logger info.😖 

![)6)H WRK7EVWMBBDTCFV 8](https://github.com/huggingface/transformers/assets/43177954/e8e7294c-ef53-4265-a486-5401b56db14a)


> Thanks for updating this and contributing to improving the code!
> 
> For the failing (unrelated) tests, there was a fix merge into main recently - #28202. Could you rebase and push the updated branch? This should resolve and trigger another CI run

@amyeroberts It works！It seems the workflow need extra approval to finish merge.

@izyForever Yes - the doc tests aren't automatically run for security reasons. I've approved the workflow now - once that passes I'll merge in. 

Thanks again for your contribution!
ImportError: cannot import name 'ESMForMaskedLM' from 'transformers'
Author: johnnytam100
Pull request: no
Labels: name: bug

### System Info

```shell
Hi, I am trying to use the esm model for protein sequence embeddings on Colab notebook:

1) I installed transformers with torch
!pip install transformers[torch]

2) Follow the example here: https://huggingface.co/facebook/esm-1b
from transformers import ESMForMaskedLM, ESMTokenizer
tokenizer = ESMTokenizer.from_pretrained("facebook/esm-1b", do_lower_case=False )
model = ESMForMaskedLM.from_pretrained("facebook/esm-1b")
sequence_Example = "QERLKSIVRILE"
encoded_input = tokenizer(sequence_Example, return_tensors='pt')
output = model(**encoded_input)

3) And got the import error:

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-7-8f12e906d421> in <module>()
----> 1 from transformers import ESMForMaskedLM, ESMTokenizer
      2 tokenizer = ESMTokenizer.from_pretrained("facebook/esm-1b", do_lower_case=False )
      3 model = ESMForMaskedLM.from_pretrained("facebook/esm-1b")
      4 sequence_Example = "QERLKSIVRILE"
      5 encoded_input = tokenizer(sequence_Example, return_tensors='pt')

ImportError: cannot import name 'ESMForMaskedLM' from 'transformers' (/usr/local/lib/python3.7/dist-packages/transformers/__init__.py)

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------

Any idea how to solve this? Thanks in advance.
```


### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

(as described)

### Expected behavior

```shell
(as described)
```


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hello, did you manage to solve it? I got the same issue

No... No one helped...

Hi, I found this over on hugging face
https://discuss.huggingface.co/t/solved-model-esm-1b-is-not-defined/17104


> Hi, I found this over on hugging face https://discuss.huggingface.co/t/solved-model-esm-1b-is-not-defined/17104

This doesn't work either. 
So why ESM can't be used with transformers even ESM2 has been published and ESM is just on the main page of this github.

It's `EsmForMaskedLM` not `ESMForMaskedLM`: https://huggingface.co/docs/transformers/model_doc/esm#transformers.EsmForMaskedLM
Needed disk space of RAG model
Author: wangjinghao123
Pull request: no

### System Info

Hi, I'm trying to finetune RAG model. I have downloaded the wiki_dpr dataset in advance, but during the finetuning the following occured.

Loading index from wiki_dpr with index name exact
WARNING:datasets.builder:Using custom data configuration psgs_w100.nq.exact-f44336eb78506dd9
Traceback (most recent call last):
  File "transformers/examples/research_projects/rag/finetune_rag.py", line 649, in <module>
    main(args)
  File "transformers/examples/research_projects/rag/finetune_rag.py", line 586, in main
    model: GenerativeQAModule = GenerativeQAModule(args)
  File "transformers/examples/research_projects/rag/finetune_rag.py", line 195, in __init__
    self.model.retriever.init_retrieval(self.distributed_port)
  File "/home/lr/jhwang/wiki_auto_split/transformers/examples/research_projects/rag/distributed_pytorch_retriever.py", line 71, in init_retrieval
    self.index.init_index()
  File "/home/lr/jhwang/.local/lib/python3.6/site-packages/transformers/models/rag/retrieval_rag.py", line 284, in init_index
    dummy=self.use_dummy_dataset,
  File "/home/lr/jhwang/.local/lib/python3.6/site-packages/datasets/load.py", line 1751, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/lr/jhwang/.local/lib/python3.6/site-packages/datasets/builder.py", line 651, in download_and_prepare
    f"Not enough disk space. Needed: {size_str(self.info.size_in_bytes or 0)} (download: {size_str(self.info.download_size or 0)}, generated: {size_str(self.info.dataset_size or 0)}, post-processed: {size_str(self.info.post_processing_size or 0)})"
OSError: Not enough disk space. Needed: 174.51 GiB (download: 66.09 GiB, generated: 73.03 GiB, post-processed: 35.39 GiB)

Is this normal? How many GBs does the RAG model need in total? Also, is there a way to change the path so that I can download the 174.51GB data to another place? Thanks a lot!

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

CUDA_VISIBLE_DEVICES="0" python3 transformers/examples/research_projects/rag/finetune_rag.py \
    --data_dir wiki_auto \
    --output_dir wiki_auto_rag \
    --model_name_or_path facebook/rag-sequence-base \
    --model_type rag_sequence \
    --fp16 \
    --gpus 1\

### Expected behavior

finetune process

Comments:
Hi @wangjinghao123, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
remove the assumption: source and target vocab should be shared for Marian model
Author: Kitty-Bert-Mom
Pull request: yes

# Motivation
I faced below errors when run https://github.com/huggingface/transformers/blob/b880508440f43f80e35a78ccd2a32f3bde91cb23/src/transformers/models/marian/convert_marian_to_pytorch.py
RuntimeError: Error(s) in loading state_dict for MarianMTModel:
**size mismatch** for final_logits_bias: copying a param with shape **torch.Size([1, 52])** from checkpoint, the shape in current model is **torch.Size([1, 36]).**
size mismatch for model.decoder.embed_tokens.weight: copying a param with shape **torch.Size([52, 256])** from checkpoint, the shape in current model is torch.Size(**[36, 256]).**
You may consider adding ignore_mismatched_sizes=True in the model from_pretrained method.

# Root cause

for my C++ marian model, source vocab has 36 words. the target vocab has 52 words. Current transfomer marian model code treats the source vocab same as the target vocab. That's why the errors said copying a param with shape torch.Size([52, 256]) from checkpoint, the shape in current model is torch.Size([36, 256]).

I saw the comment in the marianMTModel, there is an assumption for the model, that the source and target vocab should be shared. But the assumption is not suitable for all marian models. at least the model I trained is not shared. (I can not retrain the model from scratch due to some reason)

# What does this PR do?
 remove the assumption: source and target vocab should be shared for Marian model

## Was this discussed/approved via a Github issue?
https://github.com/huggingface/transformers/issues/26338
https://github.com/huggingface/transformers/issues/15109


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@ArthurZucker @younesbelkada @gante

Comments:
cc @ArthurZucker I believe you wanted to dive into this in the days/weeks to come

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Translate configuration.md to chinese
Author: ji-huazhong
Pull request: yes

# What does this PR do?
Part of https://github.com/huggingface/transformers/issues/26803


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->
cc @stevhliu


Comments: