noamaanMulla-03/transformers-issues huggingface dataset:
[WIP] New model RTDetr
Author: rafaelpadilla
Pull request: yes
Labels: name: New model

# What does this PR do?

Adds new model RTDetr. 

Image processing:
- [x] preprocess 
- [x] post_process
- [x] post_process_object_detection

Tests:
- [x] image processing: test_image_processor_outputs
- [x] image processing: test_multiple_images_processor_outputs 
- [x] model: logits and boxes match the original model
- [ ] model: unit tests for `modeling_rt_detr.py` are passing

Backbone:
- [x] adjust backbone to be compatible with Timm
- [x] convert backbone weights to be compatible with Timm 

General:
- [x] review docstrings
- [x] check variable names
- [x] check order of classes

Fixes #26742 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26939). All of your documentation changes will be reflected on that endpoint.
Enrich TTS pipeline parameters naming
Author: ylacombe
Pull request: yes

# What does this PR do?

#26369 highlighted that the use of `forward_params` in the TTS pipeline was not clear enough. This PR enrich a bit the docstrings to correct this oversight.

Note that following a discussion with @Vaibhavs10, @sanchit-gandhi and @Narsil in the same issue, I came to the conclusion that:
- adding `max_new_tokens` would add confusion to the pipeline usage, since generative and non-generative models coexist
- in the same manner, I believe that renaming `forward_params` to `generate_params` would be equally as confusing

As @Narsil [noted](https://github.com/huggingface/transformers/issues/26369#issuecomment-1736275171), an user that would like to use advanced parameter controls would probably use the classic transformers usage (tokenizer + model + etc).

Of course, I'm open to discussion and modification of current behavior if this problem recurs in the future.

<!-- Remove if not applicable -->

Fixes #26369


## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).



## Who can review?
Hey @osanseviero, @Narsil and @ArthurZucker ! Do you feel like this resolve the issue for now?

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Gentle ping here @ylacombe @sanchit-gandhi ?
I'd like to promote more complex operations and the ability to play around with pipelines for TTA/S. Would be cool to be able to do it via `generate_kwargs` to showcase similarity across various pipeline usage.

Hey @Vaibhavs10, @sanchit-gandhi and @ArthurZucker, I've followed your advice and made it compatible with `generate_kwargs` if the model is generative.  `forward_params` is still usable for both type of models but generate_kwargs has priority over `forward_params` if usable.

Let me know your opinion on it!

The MusicGen TTA doctest is currently timing out (> 120s). Given we already set a low generation max length (35 tokens), I don't think we can really reduce the time for this test much further. Do you think it makes sense to switch to using the VITS model on the doctest, since it'll run in <10s?

Thanks for the reviews here @sanchit-gandhi and @ArthurZucker, I have updated according to your comments, and will merge once the checks are done!

Looks like the doc tests might be too slow still: https://app.circleci.com/pipelines/github/huggingface/transformers/75458/workflows/bbc191e3-5c49-410e-a720-307229af6d37/jobs/957271

Should we use a different checkpoint? https://github.com/huggingface/transformers/pull/26473#issuecomment-1759485638

Hey @sanchit-gandhi, this slips my mind, looking at this in about an hour ;) 

No timing out anymore, just by syncing the branch with main! Is it okay to merge in that case @amyeroberts and @sanchit-gandhi ?

If tests are passing and you have core maintainer approval (from @ArthurZucker here) you're good to go!
Wrap `_prepare_4d_causal_attention_mask` as a leaf function
Author: michaelbenayoun
Pull request: yes

# What does this PR do?

This wraps the `_prepare_4d_causal_attention_mask` as a FX leaf function for similar reasons than [here](https://github.com/huggingface/transformers/pull/26414#issuecomment-1737443846).

The only consequence it has is that it will not be possible to edit this function by using `torch.fx`. It is not a big deal at all, but I will remove this constraint as soon as possible.

Comments:
No it needs to happen at the top-module level.

It is, the only difference it implies is that it will not be possible to edit this function via `torch.fx`. 

_The documentation is not available anymore as the PR was closed or merged._

Actually I have a workaround that works for my purposes in `optimum-neuron`, I can reverse the changes or keep them like that, as you wish. It's not a big deal in any case.

I think it's fine to leave as-is - people might want to use torch tracing outside of `optimum-neuron`.
Removed the redundant SiLUActivation class.
Author: hisushanta
Pull request: yes

Removed the redundant **SiLUActivation** class and now use **nn.SiLU** directly.

See now what looks like:
![Screenshot (7)](https://github.com/huggingface/transformers/assets/93595990/c2b33b4b-4028-4b01-92fd-daead767e174)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?





Comments:
Hi! I first set RUN_SLOW=1 and then ran the PyTest command to verify that all of my tests passed successfully.


The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27136). All of your documentation changes will be reflected on that endpoint.
VideoMAE missing CLS tokens in embedding
Author: z5163449
Pull request: no

### System Info

I'm not sure if I've missed something in the code, but I can't seem to find where the CLS tokens are added? I have input data of shape (64,45,2,32,32) with tubelet size = 5, patch_size = 4. This results in a sequence length of 576. From my understanding that is the total number of tubelets. I see that after the data is passed through the embedding layer the final embedding shape is (64,576,768) where 768 is the hidden size. However, should the dimensions not be (64,577,768) since we should be adding a CLS token to the sequence?

Would be great to get hear back soon because I'm not sure if I'm wrong or if there is something wrong with the code.

Thanks!
@NielsRogge 

### Reproduction

pixel_values = torch.randn(1,45, 2, 32, 32)
config = VideoMAEConfig()
config.num_frames = 45
config.image_size = 32
config.patch_size = 4
config.tubelet_size = 5
config.num_channels = 2

num_patches_per_frame = (model.config.image_size // model.config.patch_size) ** 2
seq_length = (model.config.num_frames // model.config.tubelet_size) * num_patches_per_frame
print(seq_length.shape)

videomae = VideoMAEModel(config)
output = videomae(pixel_values, output_hidden_states=True)
sequence_output = output[0]
print(sequence_output.shape)

### Expected behavior
seq_length = 576
sequence_output = (1,577,768)
The embedding sequence length should be total number of  tubelets + 1

Comments:
Hi,

VideoMAE doesn't use a CLS token, so this can be fixed in the docstring. The number of tokens sent through the Transformer equals (number of frames // tubelet_size) * (height // patch_size) * (width // patch_size).

For video classification, the authors average pool the final hidden states of the tokens before applying a final classification head.

Do you mind opening a PR to fix [this docstring](https://github.com/huggingface/transformers/blob/8fb4d0e4b46282d96386c229b9fb18bf7c80c25a/src/transformers/models/videomae/modeling_videomae.py#L901-L902)?

@NielsRogge Hi, sorry for coming back to this, and this may be a more general question, but why would the authors use the final hidden states of the model (that would more closely resemble the inputs again), instead of an intermediate state? I know the shapes are fixed and its not a compressing autoencoder, but why the last hidden state?

People typically use the last hidden states of Transformer-based models as features for classification layers. One of the first papers that did this was BERT.
Trainer doesn't shift labels for CAUSAL_LM PEFT models with label smoothing enabled
Author: kkteru
Pull request: no

### System Info

- `transformers` version: 4.35.0.dev0
- Platform: Linux-5.15.90.1-microsoft-standard-WSL2-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.2
- Accelerate version: 0.24.1
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): 2.14.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.0 (cpu)
- Jax version: 0.4.13
- JaxLib version: 0.4.13
- Using GPU in script?: RTX 3090
- Using distributed or parallel set-up in script?: No

### Who can help?

@pacman100, @muellerzr 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

I am not sure how to articulate this silent bug with a code snippet. I will try to explain it referring to the code and hopefully it will be clear.

The [`compute_loss`](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L2672) function in the [Trainer](https://github.com/huggingface/transformers/blob/main/src/transformers/trainer.py#L229) class gets the `model_name` from `model.base_model` when the model is a `PeftModel` as written [here](https://github.com/huggingface/transformers/blob/84724efd101af52ed3d6af878e41ff8fd651a9cc/src/transformers/trainer.py#L2690). The `base_model` of a `PeftModel` is defined [here](https://github.com/huggingface/peft/blob/884b1ac3a8ef49c9301b5bbf02e8bc64349e95f9/src/peft/peft_model.py#L119) as one of [`PEFT_TYPE_TO_MODEL_MAPPING`](https://github.com/huggingface/peft/blob/main/src/peft/peft_model.py#L68). The 'true' base model is actually stored at `base_model.model` as declared [here](https://github.com/huggingface/peft/blob/884b1ac3a8ef49c9301b5bbf02e8bc64349e95f9/src/peft/tuners/tuners_utils.py#L70).

The issue is--in [this](https://github.com/huggingface/transformers/blob/84724efd101af52ed3d6af878e41ff8fd651a9cc/src/transformers/trainer.py#L2693) line of `compute_loss` method, the check to shift labels is done by seeing if `model_name` is inside the [`MODEL_FOR_CAUSAL_LM_MAPPING_NAMES`](https://github.com/huggingface/transformers/blob/main/src/transformers/models/auto/modeling_auto.py#L384) list. Since the `model_name` for a `PeftModel` isn't in that list, the labels aren't shifted.

From what I can tell, a simple fix without breaking anything else could be to modify [this](https://github.com/huggingface/transformers/blob/84724efd101af52ed3d6af878e41ff8fd651a9cc/src/transformers/trainer.py#L2690) line to:

```
model_name = unwrap_model(model.base_model.model)._get_name()
```

### Expected behavior

The labels should be shifted for causal language modelling tasks even when using peft models.

Comments:
Created a PR for this quick fix, if that helps. 

cc @younesbelkada 

Thanks for the deepdive! I will reply you on the PR itself
Unable to load FuyuProcessor, FuyuForCausalLM from transformers
Author: akshaytheau
Pull request: no

### System Info

I tried running the code on google colab. I am using the latest transformers verison 4.34.1

@ArthurZucker 

### Who can help?

@ArthurZucker 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

Try importing FuyuProcessor and FuyuforCasualLM from transformers

### Expected behavior

It is should be imported properly.

Comments:
Hey! That's is expected, you need to use the `main` branch and install from source as t is not part of the latest release. 
`pip install git+https://github.com/huggingface/transformers`

Thanks it worked.

it didnt work for me, i m trying to run it through colab

You need to reload the kernel

> it didnt work for me, i m trying to run it through colab

Try install `torch` and `pillow`, it works for me.
```
pip install torch pillow
```
Typo of layernorm naming.
Author: RunpeiDong
Pull request: no

the `pre_layrnorm` should be `pre_layernorm`. Many of them are written wrongly, but the `post_layernorm` is all written correctly. Why is it? Should it be fixed?

Some examples of this typo:
https://github.com/huggingface/transformers/blob/4557a0dede92ce985576fac478b754d76bba3c18/src/transformers/models/clip/modeling_clip.py#L844C57-L844C57




Comments:
Hi @RunpeiDong, please see this related issue: #27190 


> Hi @RunpeiDong, please see this related issue: #27190

Thanks @vanpelt. I understand the tedious work to do if we want to address typos in previous models. I guess we can only try to avoid it in future models.
4.33.2 breaks deepspeed_custom_scheduler fixed in 4.33.1
Author: wizyoung
Pull request: no

### System Info

- `transformers` version: 4.33.2
- Platform: Linux-4.14.0_1-0-0-44-x86_64-with-glibc2.27
- Python version: 3.9.17
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.2
- Accelerate version: 0.23.0
- Accelerate config:    not found
- PyTorch version (GPU?): 1.13.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@muellerzr @pacman100
4.33.1 fixed the deepspeed_custom_scheduler in https://github.com/huggingface/transformers/commit/6bc517ccd4a3bcda4d0621d54a37c3e047df223a, however, the 4.33.2 patch release broke it again.

4.33.1:
https://github.com/huggingface/transformers/blob/fa6107c97edf7cf725305a34735a57875b67d85e/src/transformers/trainer.py#L2367-L2380
vs broken 4.33.2:
https://github.com/huggingface/transformers/blob/6da93f5580e109fad5f7b523cf2b6e8a5bafb623/src/transformers/trainer.py#L2364-L2370

4.33.1:
https://github.com/huggingface/transformers/blob/fa6107c97edf7cf725305a34735a57875b67d85e/src/transformers/trainer.py#L2434-L2445
vs broken 4.33.2:
https://github.com/huggingface/transformers/blob/6da93f5580e109fad5f7b523cf2b6e8a5bafb623/src/transformers/trainer.py#L2424-L2431

In version 4.33.2, if we use HF lr_scheduler and have deepspeed enabled, the lr_scheduler cannot be saved and loaded normally. This is a huge affect that breaks large scale training-pause-resuming loop. Considering 4.33.2 fixes two critical issues, I suggest release a new patch as this bug affects more people.

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

100%

### Expected behavior

A new patch release.

Comments:
Weird... The github 4.33.1 tag fixed the bug as I mentioned above, but the pypi release 4.33.1 src code still lies the bug, which is inconsistent.

Hello @wizyoung, this should be fixed in https://github.com/huggingface/transformers/releases/tag/v4.33.3
Allow phi 1.5 to accept attention mask as a parameter
Author: bnicholl
Pull request: no

### Feature request

As of right not, phi-1.5 does not accept attention masking for training or running inferences in batches. Could we get phi-1.5 to accept attention mask as a parameter

### Motivation

I would like attention masking for fine tuning

### Your contribution

If this isn't planned on getting taken care of, I could begin incorporating the weights into a torch module that will accept attention mask myself. 

Comments:
Hey @bnicholl, phi 1.5 is being added to the library here: https://github.com/huggingface/transformers/pull/26170

This will support the attention mask as a parameter.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
embedding_size=0 when training with deepspeed zero3
Author: iMountTai
Pull request: no
Labels: name: solved

### System Info

![image](https://github.com/huggingface/transformers/assets/35353688/5abbcf33-44cd-475c-8055-85e910730127)


### Who can help?

@ArthurZucker @pacman100 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

lora training with deepspeed zero3
```
model.get_input_embeddings().weight.shape[0] = 0
```
with Transformers4.31.0
```
model.get_input_embeddings().weight.shape[0] = 0
model.resize_token_embeddings(55296)
model.get_input_embeddings().weight.shape[0] = 55296
```
with Transformers4.33.0.dev0
```
model.get_input_embeddings().weight.shape[0] = 0
model.resize_token_embeddings(55296)
model.get_input_embeddings().weight.shape[0] = 0
```

### Expected behavior

embedding_size is returned for a correct value, otherwise
![image](https://github.com/huggingface/transformers/assets/35353688/3a3aa27e-e4af-4e18-ac65-278cf29ef34f)


Comments:
Hey! Thanks for opening an issue, could you write a bit more about the issue you have? I'm sorry but I can't really understand the problem as is. Do you have a reproducer?

related to this issue https://github.com/huggingface/transformers/issues/25977

Hello, please clearly explain the issue along with a minimal example to reproduce it.

Hello, this has been fixed, could you try to run using the main branch?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
testing doc-builder
Author: mishig25
Pull request: yes

testing https://github.com/huggingface/doc-builder/pull/423


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_26582). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
[`PEFT` / `Tests` ] Fix peft integration failing tests
Author: younesbelkada
Pull request: yes

# What does this PR do?

As per title, this PR fixes issues with respect to failing PEFT tests.
Due to the safetensors format being the default format of saving models we need to adapt expected results from tests accordingly. Also added regression tests to make sure previous behaviour is preserved

Links to failing jobs: https://github.com/huggingface/transformers/actions/runs/6727555396/job/18285707568 / https://github.com/huggingface/transformers/actions/runs/6727555396/job/18285709440

cc @amyeroberts 


Comments:
_The documentation is not available anymore as the PR was closed or merged._
[docs] Custom model doc update
Author: MKhalusova
Pull request: yes

This PR adds a documentation update to address the feedback left [here](https://huggingface.co/jinaai/jina-embeddings-v2-base-en/discussions/5#65380596ffe3e0513108cd3a), quote:

```
The clarity of the transformers documentation on custom models could be improved by describing 
the syntax of auto_map in config.json. We never used the register functions and would just directly 
modify the config.json. The behaviour that allowed one to use code on the Hub from another repo 
using "--" doesn't seem to be documented anywhere but we figured out that it was possible because 
save_pretrained saves in this format when using a custom model. The feature does seem to be pretty 
new though (I believe ~6 months ago?) so maybe that is why it hasn't been too well documented yet. 
But we think that if it was better communicated to users that it was possible to do this, more people 
would develop on the Hub as we did.
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._