noamaanMulla-03/transformers-issues huggingface dataset:
[Fuyu] Add tests
Author: NielsRogge
Pull request: yes

# What does this PR do?

Fuyu currently is not being tested. This PR makes sure the model is tested.

To do:

- [ ] the model should ideally work out-of-the-box with the `image-to-text` and `vqa` pipelines, so processor + image processor should be battle tested to make sure they fit the API. => can be addressed in a separate PR

For the latter, the following should ideally work without any additional code:

```
from transformers import FuyuProcessor, FuyuForCausalLM

processor = FuyuProcessor.from_pretrained("adept/fuyu-8b")
model = FuyuForCausalLM.from_pretrained("adept/fuyu-8b")

inputs = processor(images=image, text=text, return_tensors="pt")

outputs = model.generate(**inputs)
predictions = processor.batch_decode(outputs[0], skip_special_tokens=True)
```
cc @molbap 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

> Thanks for adding this!
> 
> Improvements on the modeling file are great. For the tests, lets coordinate with @molbap to figure the best sequence of merges as there's a few pending PRs relating to Fuyu that are due to be merged e.g. #27133 #27007 #27083 and might affect this PR

@NielsRogge, I think once you remove the extra file `src/transformers/models/fuyu/test.py`, and when CI green, we can request a final review.

We have to put `src/transformers/models/fuyu/modeling_fuyu.py` in `utils/slow_documentation_tests.txt` to avoid timeout on CircleCI
[Table Transformer] Add Transformers-native checkpoints
Author: NielsRogge
Pull request: yes

# What does this PR do?

This PR adds Table Transformer checkpoints that leverages the `AutoBackbone` API (instead of the timm library) for the ResNet backbones. This enables people to use the `TableTransformer` models without relying on timm as dependency.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Update spelling mistake
Author: LimJing7
Pull request: yes

# What does this PR do?
thoroughly was misspelled thouroughly

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@ArthurZucker @younesbelkada

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
dead link to vitdet-base-patch16-224
Author: dotneet
Pull request: no

This link is dead.

[google/vitdet-base-patch16-224](https://huggingface.co/google/vitdet-base-patch16-224)
https://github.com/huggingface/transformers/blob/897a826d830e8b1e03eb482b165b5d88a7a08d5f/src/transformers/models/vitdet/configuration_vitdet.py#L35

Comments:
@dotneet Hey,  I want to work on this issue. Can you assign to me ?

First come, first served @Ankit8848, feel free to open a PR.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

 @Ankit8848 Is this bug fixed? I can not load the 'google/vitdet-base-patch16-224' model. Thx for your help.

```
from transformers import VitDetConfig, VitDetModel
model = VitDetModel.from_pretrained("google/vitdet-base-patch16-224")
```

@betterze i don't know, I have not worked on this issue.
Add pop2piano
Author: ArthurZucker
Pull request: no
Labels: name: New model

### Model description

- Introduce a large amount of paired and synchronised {pop, piano cover} data using an automated pipeline.
- Pop2Piano, a Transformer network that generates piano covers given waveforms of pop music. 
- First model to directly generate a piano cover from pop audio without melody and chord extraction modules.
- Uses a T5  model so should be straightforward. 

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

- Weights : https://github.com/sweetcocoa/pop2piano/releases/download/dpi_2k_epoch/model-1999-val_0.67311615.ckpt 
- Code : https://github.com/sweetcocoa/pop2piano/
- Paper : https://arxiv.org/abs/2211.00895
- Colab : https://colab.research.google.com/drive/1rBAs2TkryDnnQOhcM-mtlrgtL2h3ekml?usp=sharing 

Comments:
Hi @ArthurZucker could you please share the progress of the model addition?(I am asking because the last time this Issue has had any actions was in Nov 8, 2022). I tried this model on colab and really loved it. I want to add / help adding this model to HF. Is it possible that we can collaborate in this addition?  

Hey! I did not start at all! Feel free to open a PR and ping me for pointers/help! I won't have time to do it alone but would love to collaborate ! 😉 


@ArthurZucker Ok, my term exams are till 3rd March so I will start working from 4th, in meantime I will open a PR. Do you want to continue communicating through that PR or how about communicating through slack/discord if it's possible? 

Sure, will invite you to slack if you can share your email, arthur@hf.co! 

> Sure, will invite you to slack if you can share your email, [arthur@hf.co](mailto:arthur@hf.co)!

My email is - susnatodhar10@gmail.com

@ArthurZucker 

Just invited you! Good luck on your mid terms 😉 

Also if anyone want to tackle this before, ping me and will add you to the channel

@ArthurZucker How to retrain model and how to get datasets to retrain? ! Help me
[`CircleCI`] skip test_assisted_decoding_sample for everyone
Author: ArthurZucker
Pull request: yes

# What does this PR do?
cc @gante I'm having a look at why this is failing but skipping for now. (standalone works well) 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@ArthurZucker [this PR](https://github.com/huggingface/transformers/pull/27503) should fix it!

I decided to rely on the tests instead of asking the contributor to double-check, but forgot that the tests were stochastic -- they rely on `sample`. The PR that caused this crash had a lucky CI run, so the problem got merged while undetected.

In other words, I should have been more careful :)
[ `PretrainedConfig`] Improve messaging 
Author: ArthurZucker
Pull request: yes

# What does this PR do?
Fixes #26720 by improving the error message printed out when someone tries:
```python 
>>> from transformers import AutoModel
>>> AutoModel.from_pretrained(".gpt2")
...
OSError: Incorrect path_or_model_id: '.gpt2'. Please provide either the path to a local folder or the repo_id of a model on the Hub. Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: '.gpt2'.
```
And: 
```python 
>>> from transformers import AutoConfig
>>> AutoConfig.from_pretrained(".gpt2")
...
OSError: Incorrect path_or_model_id: '.gpt2'. Please provide either the path to a local folder or the repo_id of a model on the Hub. Repo id must use alphanumeric chars or '-', '_', '.', '--' and '..' are forbidden, '-' and '.' cannot start or end the name, max length is 96: '.gpt2'.
```

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Sure I'll add a test! 😈
Fix wav2vec2 params
Author: muellerzr
Pull request: yes

# What does this PR do?

Caught on the Accelerate CI, the `wav2vec2` script doesn't take in `fp16` so removes it as a test arg. 

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@amyeroberts 


Comments:
Perf torch compile
Author: jiaqiw09
Pull request: yes

# What does this PR do?

Part of  #26803 

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? _not necessary_


## Who can review?
@stevhliu

Comments:
@stevhliu

Hi,

here is another PR.

Best

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27422). All of your documentation changes will be reflected on that endpoint.
translate hpo_train.md and perf_hardware.md to chinese
Author: jiaqiw09
Pull request: yes

# What does this PR do?

Part of  #26803 

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests? _not necessary_


## Who can review?
@stevhliu

Comments:
@stevhliu

Hi,

Here is another pr. Thanks for my DIY Computer experience， or I can not understand what the docs talking about.

I think I will be free next couple of days to translate some "difficult but practical“ docs.

Best

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27431). All of your documentation changes will be reflected on that endpoint.

@stevhliu 

hi, thansk for your review. I just fix problems mentioned in reviews.

Best
Translating `en/model_doc` docs to Japanese.
Author: Yuki-Imajuku
Pull request: yes

# What does this PR do?
translating `en/model_doc` docs to Japanese.

Fixes #27392 

## Before submitting

- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?
@stevhliu and @MKhalusova 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_27401). All of your documentation changes will be reflected on that endpoint.
Fixing the failure of models without max_position_embeddings attribute.
Author: AdamLouly
Pull request: yes

**Changes Made**

Added handling for configurations that may not have the max_position_embeddings attribute.
Introduced a default value of 1024 for max_position_embeddings when it's missing in the configuration.

**Motivation and Context**

Fixing this issue: https://github.com/huggingface/transformers/issues/27498

This PR addresses an issue where some configurations lack the max_position_embeddings attribute, causing failures in certain scenarios.

Comments:
@amyeroberts  when can we merge this? I have found another issue and we need this to be merged first before fixing the other one.

@AdamLouly - we can merge now :)
Count of tokens seen during training in Trainer
Author: jpgard
Pull request: no
Labels: name: Feature request

### Feature request

The `Trainer` API should track and log the number of tokens seen during training.

While it sometimes could (maybe?) be possible to back out the number of tokens seen from the FLOS, or by iterating over the whole dataset, it would make a lot of sense for the Trainer API to track the number of tokens seen (and it shouldn't be necessary to completely iterate over a model's training loop just to compute the count of tokens, which is the only current implementation of any token-related metric in Trainer, [`Trainer.num_tokens()`](https://github.com/huggingface/transformers/blob/acc394c4f5e1283c19783581790b3dc3105a3697/src/transformers/trainer.py#L1180)).

This can't currently be implemented in a CallBack, because callbacks don't have access to the training data (only the trainer state).

### Motivation

Number of tokens seen is an essential metric tracked in nearly every LLM training run. It is widely considered one of the fundamental drivers of model quality (tokens seen during training is reported for nearly every major LLM release). It seems that any language model developer using Hugging Face would like to know this metric for their training runs -- it maybe even more important and useful than the FLOS, and perhaps as important as the number of gradient steps. 

In any case, it's an extremely useful number to have, and it must be tracked during training as the model consumes examples.

### Your contribution

I'm willing to contribute this but would like some guidance on the overall design first.

In particular, here's what I think a reasonable implementation would include:

- Add a `global_tokens_seen` or similar to the `TrainerState`. This would add only a single integer value to the `TrainerState`.
- Increment this during `Trainer._inner_training_loop()`
- Probably add this information to the logging outputs

What do the folks at HF think about that?

Comments:
+1 
I think we need this feature 

cc @muellerzr seems nice if we can make it efficient! 

Is the `tokens_per_second` we already have as part of https://github.com/huggingface/transformers/pull/25858 enough? Otherwise we can definitely add it :) 

Yeah, tokens/sec doesn't cut it for many use cases (although it is still very useful!!) -- similar to how tracking steps/sec doesn't obviate the need for a global step count.

If you can add it that would be amazing, I am sure this would be a useful feature to almost anyone training a language model. And I think there are some subtleties to how to make it work right in a distributed setting that you would probably be much better at handling....

agree tokens/sec/gpu is useful, but it fails to track `pad` tokens and if we were to do `SFTTrainer` with packing set to False, this number can be way off. So, we need a feature that tracks actual tokens seen.

thanks @muellerzr !!
Incorrect setting for num_beams in translation and summarization examples
Author: Rocketknight1
Pull request: yes

Our translation and summarization examples use `None` as a default value for `num_beams`, but this is no longer valid now that we use `GenerationConfig`. This PR fixes the examples, and also adds a guard in `GenerationConfig` so that `None` values there are replaced with `1` (and a warning is thrown).

Also, while I'm fixing example issues, some of our TF examples use `main_process_first` - this is a Torch function, and is not necessary because TF handles distributed training very differently. Since it looks like it could cause problems depending on whether Torch is also present, I'm removing it from all of the TF examples.

Fixes #27505

Comments:
_The documentation is not available anymore as the PR was closed or merged._
DeltaLM
Author: jcmc00
Pull request: no
Labels: name: New model

### Model description

DeltaLM is a multilingual encoder-decoder architecture that regards the decoder as the task layer of off-the-shelf pre-trained encoders. This architecture introduces an interleaved decoder, which has a more consistent structure with the encoder. Weights from pre-trained multilingual encoders are used to initialise both the encoder and decoder models before training on monolingual and bilingual data.

As of September 2021 DeltaLM ranks first on the [WMT21 multilingual translation task](http://www.statmt.org/wmt21/large-scale-multilingual-translation-task.html).

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation
The model implementation is available at: 
https://github.com/microsoft/unilm/tree/master/deltalm

Model weights are available:
[DeltaLM-base](https://deltalm.blob.core.windows.net/deltalm/deltalm-base.pt)
[DeltaLM-large](https://deltalm.blob.core.windows.net/deltalm/deltalm-large.pt)

Who are the authors:
@shumingma @gitnlp

I'd be happy to try work on contributing the model.

Comments:
Any progress on this?

Hi,

I've noticed there are some DeltaLM available in the Hub:
- https://huggingface.co/IDEA-CCNL/Randeng-Deltalm-362M-En-Zh
- https://huggingface.co/IDEA-CCNL/Randeng-Deltalm-362M-Zh-En

The code they use seems to be here: https://github.com/IDEA-CCNL/Fengshenbang-LM/tree/main/fengshen/models/deltalm

However, I'd be interested in using the original checkpoints available in the official repository: https://github.com/microsoft/unilm/tree/master/deltalm

Do you have any idea on how to do it?

Thanks!

It seems like DeltaLM is still the best multilingual NMT.
How can we progress?

The modeling files have already been added for this model on the hub repo e.g. [here](https://huggingface.co/IDEA-CCNL/Randeng-Deltalm-362M-En-Zh/blob/main/modeling_deltalm.py). 

At the moment, the [model config](https://huggingface.co/IDEA-CCNL/Randeng-Deltalm-362M-En-Zh/blob/main/config.json) is missing the auto_map parameter e.g. [like this one](https://huggingface.co/01-ai/Yi-34B/blob/58cb0a1e67e71efee59c3b9956723f87e97fe7b7/config.json#L5).

Once added, you should be able to use the models directly using: 

```py
from transformers import AutoMode

model = AutoModel.from_pretrained("IDEA-CCNL/Randeng-Deltalm-362M-En-Zh", trust_remote_code=True)
```

If you want to use the already implemented architecture with other checkpoints, you can upload these to the hub and point to that code in the model config using the `_name_or_path` parameter.
Fix bug for T5x to PyTorch convert script with varying encoder and decoder layers
Author: JamesJiang97
Pull request: yes

When I try to use [this script](https://github.com/huggingface/transformers/blob/main/src/transformers/models/t5/convert_t5x_checkpoint_to_pytorch.py) to convert ByT5 T5x model to PyTorch model，
I get the following error :
```
Traceback (most recent call last):
  File "/home/jiang/t5x_home/transformers/src/transformers/models/t5/convert_t5x_checkpoint_to_pytorch.py", line 231, in <module>
    convert_t5x_checkpoint_to_pytorch(
  File "/home/jiang/t5x_home/transformers/src/transformers/models/t5/convert_t5x_checkpoint_to_pytorch.py", line 200, in convert_t5x_checkpoint_to_pytorch
    load_t5x_weights_in_t5(model, config, t5x_checkpoint_path, is_encoder_only)
  File "/home/jiang/t5x_home/transformers/src/transformers/models/t5/convert_t5x_checkpoint_to_pytorch.py", line 180, in load_t5x_weights_in_t5
    converted = convert_t5x_to_pytorch(variables, num_layers=config.num_layers, is_encoder_only=is_encoder_only)
  File "/home/jiang/t5x_home/transformers/src/transformers/models/t5/convert_t5x_checkpoint_to_pytorch.py", line 117, in convert_t5x_to_pytorch
    layer_norm = t5x_layer_norm_lookup(old, i, "decoder", "pre_self_attention_layer_norm")
  File "/home/jiang/t5x_home/transformers/src/transformers/models/t5/convert_t5x_checkpoint_to_pytorch.py", line 69, in t5x_layer_norm_lookup
    return params[f"{prefix}/layers_{i}/{layer_name}/scale"]
KeyError: 'decoder/layers_4/pre_self_attention_layer_norm/scale'
```
I believe this is because the script did not distinguish between the number of decoder layers and encoder layers, and in the ByT5 model, the number of decoder layers is different from that of encoder layers. I fixed this bug by passing a 'num_decoder_layers' parameter to the relevant functions.

Comments: