noamaanMulla-03/transformers-issues huggingface dataset:
Difference between FlaxViTModel and FlaxCLIPVisionTransformer
Author: ashors1
Pull request: no

Hi, I noticed that HuggingFace has two different Flax-based implementations of the vision transformer, the [FlaxCLIPVisionTransformer](https://github.com/huggingface/transformers/blob/main/src/transformers/models/clip/modeling_flax_clip.py#L535) and the [FlaxViTModule](https://github.com/huggingface/transformers/blob/main/src/transformers/models/vit/modeling_flax_vit.py#L507). Is there a reason there is a separate ViT implementation for CLIP? Are there any notable differences between the FlaxCLIPVisionTransformer and the FlaxViTModel? Thank you!

Comments:
Please use the [forums](https://discuss.huggingface.co/) for questions like this as we keep issues for bugs and feature requests only.

I think they're equivalent (one would need to check they are both applying pre-norm or post-norm etc.), but the reason we have 2 different implementations is because Transformers has a [one model, one file philosophy](https://huggingface.co/blog/transformers-design-philosophy).
multi gpu training.
Author: iamnmn9
Pull request: no

### Feature request

What we can do if using run_clm.py to train with multi gpu's?


### Motivation

multi gpu training for faster training.

### Your contribution

can we add this in the script?
model = nn.DataParallel(model,device_ids=[i for i in range(torch.cuda.device_count())])

Comments:
If you have multiple GPUs it will run on multiple GPUs.

![image](https://user-images.githubusercontent.com/41872440/216619913-99568f8c-8a82-42ab-ae7a-dd56846dd395.png)
I have 8 gpu's in this machine.
![image](https://user-images.githubusercontent.com/41872440/216619996-7af22496-7fb8-4b6d-90f2-3606846b3cdc.png)
I think its not taking all 8 gpu's. Already tried changing batch_sizes and with multiple of 8
Added model resources for LayoutLM Issue#19848
Author: avisinghal6
Pull request: yes

Added resources to documentation of LayoutLM model as per Issue#19848.
@stevhliu 



Comments:
_The documentation is not available anymore as the PR was closed or merged._

It looks like you might've only updated your branch with the latest changes on the `main` Hugging Face repo, I don't see any of the changes! 🙈 

Hi @stevhliu, you were right, i only updated my branch (sorry my bad ), i  think i have committed the changes now. Sorry for this.

Awesome, thank you so much for your contribution! 🤗  Everything looks good to me, pinging @sgugger for a final look!
![image](https://user-images.githubusercontent.com/41872440/216619913-99568f8c-8a82-42ab-ae7a-dd56846dd395.png)
Author: iamnmn9
Pull request: no

![image](https://user-images.githubusercontent.com/41872440/216619913-99568f8c-8a82-42ab-ae7a-dd56846dd395.png)
I have 8 gpu's in this machine.
![image](https://user-images.githubusercontent.com/41872440/216619996-7af22496-7fb8-4b6d-90f2-3606846b3cdc.png)
I think its not taking all 8 gpu's. Already tried changing batch_sizes and with multiple of 8

_Originally posted by @namanpundir in https://github.com/huggingface/transformers/issues/21407#issuecomment-1415898415_
            

Comments:
Please do not spam the repository by opening duplicate issues. You can find help to debug your training by posting on the [forums](https://discuss.huggingface.co/) as we keep issues for bugs in the library and feature requests only. You will need to share how you are launching your script for anyone to be able to help.
Parallelize LongT5
Author: JamesDeAntonis
Pull request: yes

# What does this PR do?
Adds parallelization to LongT5

Fixes # (issue)
[21396](https://github.com/huggingface/transformers/issues/21396)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker and @younesbelkada


Comments:
_The documentation is not available anymore as the PR was closed or merged._

Yes, just pass along `device_map="auto"` or `device_map="balanced"` in your call to `from_pretrained` to have the model be parallelized. It will work for training and inference.

Oh sweet! Didn't know about this.  I just tried training with the longt5 3b model using accelerate and it didn't work well; GPU0 got most of the workload and the training run crashed quickly.  I tried both "auto" and "balanced".  If I use my code it works.  I realize that I could specify my own device map, but that's pretty tedious. Is there a better way to debug this? Thanks!
OSError: Token is required (`token=True`), but no token found. You need to provide a token or be logged in to Hugging Face with `huggingface-cli login` or `huggingface_hub.login`. See https://huggingface.co/settings/tokens.
Author: ucas010
Pull request: no

### System Info

copy the [RP](https://github.com/huggingface/notebooks/blob/main/examples/language_modeling.ipynb) and got the ERROR
could you pls help me ?



### Who can help?

@ArthurZucker and @younesbelkada

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

same as official script

### Expected behavior

overcome the error

Comments:
You need to make sure to execute the cell `notebook_login()` at the beginning and pass it your token (it provides a direct link to your token pages on hf.co)

pass it your token ??? I have token but how to use it ? @sgugger 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Cache size limit for generation
Author: Natooz
Pull request: yes

# What does this PR do?


Following #20767, it adds a `cache_limit` argument for `generate` for PyTorch and TensorFlow (except xla), limiting the size of the cache (`past_key_values`).
`position_ids` is stored in `model_kwargs` for concerned models.
This is a bit above 100 lines. No big deal if you consider the maintenance effort is not worth it, this is still a simple feature that can be implemented by users by overriding model methods.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? #20767
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@gante & @sgugger 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20902). All of your documentation changes will be reflected on that endpoint.

Hey @Natooz 👋 

Thank you for the PR! Looking at the PR, it is not too complex... but given the non-existent demand, it still amounts to a terrible maintenance-per-demand ratio 🙈 Our team is small, so we have to be extremely picky.

I am afraid that I will have to reject this PR. Nevertheless, I am happy to be proved wrong, and if I see demand for this feature I will come back to this PR as a reference implementation!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Problem with MBartForConditionalGeneration and MBart50TokenizerFast
Author: sunny3
Pull request: no

### System Info

sys info:

- `transformers` version: 4.25.1
- Platform: Linux-5.4.17-2136.308.9.el8uek.x86_64-x86_64-with-glibc2.28
- Python version: 3.9.7
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.13.1+cu117 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction
```python
from transformers import MBartForConditionalGeneration, MBart50TokenizerFast

model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")
tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")
sentence_kz = "сәлем досым"
tokenizer.src_lang = "kk_KZ"
encoded_kk = tokenizer(sentence_kz , return_tensors="pt")
generated_tokens = model.generate(
    **encoded_kk,
    forced_bos_token_id=tokenizer.lang_code_to_id["ru_RU"]
)
tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)
```
### Expected behavior

I use the official example of code from https://huggingface.co/facebook/mbart-large-50-many-to-many-mmt

I just changed the abbreviations for the languages (hi_IN -> kk_KZ, fr_XX -> ru_RU), wanting to get the Russian language out, but I get english instead of russian
![image](https://user-images.githubusercontent.com/25152817/209909173-4273c2b6-d167-4de3-b33b-4c778c305c4c.png)


Comments:
cc @ArthurZucker 

Maybe due to this ?  https://github.com/huggingface/transformers/issues/20610#issuecomment-1407704129

The model should be predicting in Russian, and the only reason it is not is probably because `kk` language has a very small dataset size and was thus not trained a lot. The model and the generation process work as they function as expected for lanugages that have a bigger dataset. 

For example if you use `ja_XX` you will get `['世界の友達']` which means friend of the world (Sekai no tomodachi). 
It comes down to the fine-tuning and which pair was trained with which other.
exclude deleted files in the fixup script
Author: dtuit
Pull request: yes

# What does this PR do?

Running `make fixup` after a file is deleted on a branch causes `black` to exit with an error. 

`Error: Invalid value for 'SRC ...': Path /path/to/deleted/file' does not exist.`

This PR resolves the error by setting the `git diff` flag `--diff-filter=d` to exclude deleted files from the list of modified files.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add tutorial doc for TF + TPU
Author: Rocketknight1
Pull request: yes

This is the sidebar tutorial for training with TF + TPU, to go with [the code notebook](https://github.com/huggingface/notebooks/pull/313).

Note that the Markdown is exported straight from Notion, so some formatting will probably look very wrong - I'm working on cleanup!

Comments:
_The documentation is not available anymore as the PR was closed or merged._

@sayakpaul The spaces have been removed because the extra content after them got moved to a whole other doc!
For IterableDataset, return DataLoader using self._train_batch_size. …
Author: agossard
Pull request: yes

…This is consistent with how we generate a regular DataLoader, and leads to the correct args.per_device_train_batch_size eventually ending up on each GPU.

Fixes # 21444#issuecomment-1416252207

@sgugger 

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Avoid flaky generation sampling tests
Author: ydshieh
Pull request: yes

# What does this PR do?

Avoid the CI failure
```bash
tests/models/switch_transformers/test_modeling_switch_transformers.py::SwitchTransformersModelTest::test_beam_sample_generate_dict_output
(line 3099)  RuntimeError: probability tensor contains either inf, nan or element < 0
```

For 
```bash
tests/models/marian/test_modeling_marian.py::MarianStandaloneDecoderModelTest::test_sample_generate
(line 2482)  RuntimeError: probability tensor contains either inf, nan or element < 0
```
it's not clear what I can change in https://github.com/huggingface/transformers/blob/6c62cfb2eff095c181481d8ae86c7f836b65d2d7/tests/generation/test_utils.py#L108-L155

I changed to `logits_warper_kwargs, logits_warper = self._get_warper_and_kwargs(num_beams=2)` despite this is not beam sampling test.

Comments:
_The documentation is not available anymore as the PR was closed or merged._
BigBird attention type switching
Author: kvarekamp
Pull request: no

### System Info

transformers 4.21.2 BigBirdModel

### Who can help?

@ydshieh

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Currently BigBird switches attention type from default 'block_sparse' to 'original_full' in the forward call if it encounters a batch that contains only sequences shorter than the minimum sequence length:
https://github.com/huggingface/transformers/blob/a2a3afbc8d26d6170909365ffba6bd75e186255f/src/transformers/models/big_bird/modeling_big_bird.py#L2064

However, it never switches back. This means that the exact same (long) sequence can be encoded differently depending on whether it was preceded by a batch containing only short sequences or not.



### Expected behavior

It should probably switch back to block_sparse if it can as well

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
TFBartForSequenceClassification
Author: uglyboxer
Pull request: no
Labels: name: Good First Issue

The Tensorflow version of `BartForSequenceClassification` seems to be missing.  I've been putting a port together for another use case.  Any interest in adding it to the repo?

Comments:
can you explain it in detail.

I have been working on a project for zero-shot classification using `facebook/bart-large-mnli`.  After an initial prototype using the transformers pipeline I began to work with the underlying Bart model directly.  As our project is in Tensorflow, I soon noticed there was no equivalent of `class BartForSequenceClassification` on the TF side.  

I have stitched it together: adding the classification head and loading the weights from the torch.bin file.   If you think it would be useful, I could put up a PR add it to https://github.com/huggingface/transformers/blob/main/src/transformers/models/bart/modeling_tf_bart.py  along side of the other TF ports for Bart.

Hey @uglyboxer 👋  It is indeed missing. If you have a working implementation, we'd be interested in adding it to our library!

Cool.  I'll put something together this week.
Encoding parameter for  AutoModel.from_pretrained() module.
Author: Prasath2001
Pull request: no

### Feature request

transformers.AutoModel.from_pretrained() module allows to load pretrained models from local directories as well. The local pickle files path is an argument of the above function. These files can have different encoding types.

A parameter called 'encoding' can be added in the parameter list similar to

pandas.read_csv('path/to/csv/file',encoding='utf-8')

which takes encoding type as parameter. But this proposed encoding feature must be enabled only while loading local files and should not take parameters while loading models from other sources (like HF-Hub etc).

### Motivation

I face Unicode-Decode error while loading pickled model files from local directory. To overcome this issue, the above feature would help to avoid the unicode-decode error.

### Your contribution

Unfortunately, I can't contribute now.

Comments:
Thanks for the report.
Please provide us with a reproducing example showing how a model saved with `save_pretrained` can't be reloaded with `from_pretrained`.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.
Fix `SpeechT5ForSpeechToSpeechIntegrationTests` device issue
Author: ydshieh
Pull request: yes

# What does this PR do?

Just a torch device issue being ifxed.

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Ah yes, nice catch. I don't have merge rights though.

Could you also fix this in modeling_speecht5? I think that has the same issue. Around line 2871:

```python
        if speaker_embeddings is None:
            speaker_embeddings = torch.zeros((1, 512), device=input_values.device)
```

Thanks!


> Could you also fix this in modeling_speecht5? I think that has the same issue. Around line 2871:

Done!

> Ah yes, nice catch. I don't have merge rights though.

You have approval right :-) @hollance Then I can merge 🚀
Error when fine-tuning XLM-RoBERTa base on TF/Keras
Author: sctlin
Pull request: no

Hello,

I am trying to finetune XLM-RoBERTa for text classification on tensorflow-keras. I am using the `TFXLMRobertaForSequenceClassification` class for training 

I am using google colab GPU for finetuning. Tensoflow version is 2.9.2

en_y_pred = model.predict(en_x_test_in, batch_size=128, verbose=1)
**InvalidArgumentError: indices[2,268] = 124030 is not in [0, 50265) [[node tf_roberta_for_sequence_classification_1/roberta/embeddings/Gather (defined at /usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_tf_roberta.py:149) ]] [Op:__inference_train_function_82886]

Errors may have originated from an input operation. Input Source operations connected to node tf_roberta_for_sequence_classification_1/roberta/embeddings/Gather: In[0] tf_roberta_for_sequence_classification_1/roberta/embeddings/Gather/resource:
In[1] IteratorGetNext (defined at /usr/local/lib/python3.7/dist-packages/keras/engine/training.py:866)**

Comments:
How can I fine-tune other languages in trocr? CER over 1
Author: saeu5407
Pull request: no

### System Info

I want to do fine-tuning using trocr.
There is a problem that CER exceeds 1.
The correct answer for the dataset is one word, but it seems that the predict value is too long.

What's the problem? Should I implement the new language with the following code?

I've been learning several times, but I've found that I'm making long sentences about text.

So there is a problem that CER goes over 1.

The link below is a code that only weights other languages without fine-tuning.

The predicted characters are quite long compared to the existing characters, and I wonder if there is a problem with EOS.

https://colab.research.google.com/drive/1ovc-9aXsYKDXAfrO0FUtoiLVkXEdrz6s?usp=sharing

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

https://colab.research.google.com/drive/1ovc-9aXsYKDXAfrO0FUtoiLVkXEdrz6s?usp=sharing

### Expected behavior

CER over 1

Comments:
Hi,

Refer to this thread: https://github.com/huggingface/transformers/issues/18163. Also, please ask such questions on our [forum](https://discuss.huggingface.co/), as we'd like to keep Github issues for bugs/feature requests.

Thanks!
raise ValueError(f"Unrecognized tokenizer_type {tokenizer_type}") ValueError: Unrecognized tokenizer_type BertWordPieceCase
Author: lucasjinreal
Pull request: no

BertWordPieceCase is actually a builtin tokenizer inside magatron. But when I using `convert_megatron_gpt2_checkpoint.py` to convert. I got this error:

```
raise ValueError(f"Unrecognized tokenizer_type {tokenizer_type}")
ValueError: Unrecognized tokenizer_type BertWordPieceCase
```

Just wonder, how to make it support？ Does it also works If i using default `gpt2` tokenizer? Any ulternate in transformers interms of `BertWordPieceCase` ?

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey @jinfagang 
Can you kindly tell how did you incorporate BertWordPieceCase tokenizer into the Megatron model? I am trying to do the same.
Thanks!
Fix `PushToHubCallback` import in Share a model docs
Author: ireneisdoomed
Pull request: yes

# What does this PR do?

Fixes a typo in the Share a model docs section. The example to push a Tensorflow model to the Hub used to call the method`PushToHubCallback` from `transformers.keras.callbacks`, resulting in `ImportError`.

This PR corrects that example in all languages so that `PushToHubCallback` is imported directly from `transformers`.

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

# Who can review?
@sgugger, @stevhliu and @MKhalusova
Thank you!

Comments:
_The documentation is not available anymore as the PR was closed or merged._