noamaanMulla-03/transformers-issues huggingface dataset:
Add preprocess_logits_for_metrics Trainer param
Author: davidleonfdez
Pull request: yes

# What does this PR do?

In addition to what I told in the issue, I thought adding labels as a parameter to the preprocess function could be useful with no cost. This forced me to change the order of the accumulation of labels and logits, which creates an asymmetry with respect to the other code blocks. I could change the order in those too, but I tried to change as little as possible.

With regards to adding the computation of perplexity in the examples run_clm or run_mlm, do you mean accuracy? Maybe I'm missing something, but the way I calculate perplexity having the loss is enough. 
Moreover, those examples don't pass `compute_metrics` either to the Trainer. 

I can add the computation of accuracy to the examples, using the new parameter to precompute the argmax, although it could still slow down the examples a bit.

cc @sgugger 

Fixes #15466






Comments:
_The documentation is not available anymore as the PR was closed or merged._

Thank you! I just added the accuracy to the language modeling examples and the doc change you requested.

I haven't worked with masked language models, so please check the `run_mlm` example is ok. I think the unmasked labels get the same -100 id as the padding, but maybe there's something I'm not thinking about.

I don't think you pushed your changes, you mentioned adding content.

No, sorry, maybe I expressed it poorly, I meant that I had added the same comment originally.

Thanks for your contribution!

Suspecting this PR introduced a breakage - please see: https://github.com/huggingface/transformers/issues/15898
Update delete-dev-doc job to match build-dev-doc
Author: sgugger
Pull request: yes

# What does this PR do?

This PR fixes the `delete-dev-doc` job to use the same runner as `buld-dev-doc`. It also fixes the `build-dev-doc` job failures that happen if the `doc-build-dev` repo got an update during the time the doc is built while not doing the stash if not necessary (which was the bug #15882 was trying to fix).

In passing, it makes the commit messages a little bit better (the sha for pushed during the dev doc updates are the sha of the merges, not the commits, and in the delete job, the variable picked always returned merge).

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15891). All of your documentation changes will be reflected on that endpoint.

Were you able to fix them? I was trying to fix the `delete_doc_job` as well in https://github.com/huggingface/transformers/pull/15894/files with no success

https://github.com/huggingface/transformers/pull/15894/files#r817899504

Working on it :-)
Add Swin Transformer
Author: novice03
Pull request: yes

# What does this PR do?

This PR adds the Swin Transformer model to the repository.

Paper: [https://arxiv.org/abs/2103.14030](https://arxiv.org/abs/2103.14030)
Code: [Official code](https://github.com/microsoft/Swin-Transformer) and [timm implementation](https://github.com/rwightman/pytorch-image-models/blob/master/timm/models/swin_transformer.py)
Checkpoints: [Image classification](https://github.com/microsoft/Swin-Transformer)

Fixes #14760 

My code is modeled after the ViT implementation. There are a few tasks to do:

- [x] Fix `TypeError: Object of type type is not JSON serializable` when saving `SwinConfig`
- [x] Fix tests in test_modeling_swin.py

## Who can review?

@NielsRogge 

Comments:
Thanks for your review. I overlooked the fact the reviewer is made a co-author by GitHub. I'll keep that in mind going forward.

> I overlooked the fact the reviewer is made a co-author by GitHub. I'll keep that in mind going forward.

I'm pretty sure you didn't know, which is why I'm telling you :-)

Hi, could you please also add swin transformer for semantic segmentation? Right now, I saw it's only for image classification. Thank you.

Hi @vaneshieh, I am not sure if Swin for sematic segmentation would be feasible. This would require incorporation of other models such as UPerNet into huggingface, which may reequire a lot of extra work.

Hi, `SwinForSemanticSegmentation` should be relatively easy to add since it uses the same head (UperNet) as `BeitForSemanticSegmentation`, which I already added. Hence, it would mostly be a copy-paste, the only thing to write is the conversion script.

Oh, I wasn't aware of that. I can take this up in ~ 2 weeks if no one else does.

> Oh, I wasn't aware of that. I can take this up in ~ 2 weeks if no one else does.

Will or where will you update this after you created it? Thank you so much!

I can leave a message here after it's done.
Disabling Eager Mode Prevents Loading Pre-Trained BERT
Author: msandrewhoang03
Pull request: no
Labels: name: wontfix

name: TensorFlow

name: Core: Modeling

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...):
**BERT**
Language I am using the model on (English, Chinese ...):
**English**

## To reproduce

Steps to reproduce the behavior:

1. Disable Tensorflow eager mode `tf.compat.v1.disable_v2_behavior()`
2. Create a pretrained BERT instance `model = transformers.TFBertModel.from_pretrained("bert-base-uncased")`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
`model` contains a pre-trained BERT model

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version:
- Platform: Windows-10-10.0.18362-SP0
- Python version: 3.7.6
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.0.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Comments:
TFBertModel can auto use gpu?  i find model(np.array(input_ids)) is slow,cost 100+ms
In [14]: %timeit model_outputs = model(np.array(input_ids))
133 ms ± 3.4 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [15]: %time model_outputs = model(np.array(input_ids))
CPU times: user 330 ms, sys: 14.4 ms, total: 344 ms
Wall time: 158 ms


This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


I have a similar issue. Did you manage to find a fix for this @msandrewhoang03 ?
Fix a TF Vision Encoder Decoder test
Author: ydshieh
Pull request: yes

# What does this PR do?

Simple fix for the following CI failure: 

```
tests.vision_encoder_decoder.test_modeling_tf_vision_encoder_decoder.TFViT2GPT2EncoderDecoderModelTest.test_pt_tf_equivalence
```
This is due to a few PyTorch device issue (need `pt_inputs` to be on the correct device).

Tested on GCP GPU VM - it pass now.

@patrickvonplaten @sgugger 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15896). All of your documentation changes will be reflected on that endpoint.

Probably wait @patrickvonplaten too, but once ready, should I merge this PR myself ? (or still should be one of you to do so?)

You can merge your own PRs as long as you have the approvals you want :-)
Fix Bug in FlaxWav2Vec2 Slow Test
Author: sanchit-gandhi
Pull request: yes

This PR fixes a small bug the `test_inference_ctc_robust_batched` FlaxWav2Vec2 slow test. Currently, the input processor is run with `return_tensors="pt"` and `trunctation=True`:
https://github.com/huggingface/transformers/blob/05c237ea94e08786abbac6c6185cfdfa262a8c53/tests/wav2vec2/test_modeling_flax_wav2vec2.py#L387
However, as outlined in `feature_extraction_sequence_utils.py`, when setting `trunctation=True`, one must **also** specify the `max_len`:
https://github.com/huggingface/transformers/blob/6e57a56987ff201747f5f01bbce3ed2c0fda1910/src/transformers/feature_extraction_sequence_utils.py#L326-L327
The modifications make the change to `return_tensors="np"` and remove the `truncation` flag. The first change returns the native tensor type for Flax (`np` as opposed to `pt`), and the second change aligns the test with its PyTorch counterpart:
https://github.com/huggingface/transformers/blob/6e57a56987ff201747f5f01bbce3ed2c0fda1910/tests/wav2vec2/test_modeling_wav2vec2.py#L1170


 

Comments:
Nan when training LayoutLM_V2 Model
Author: Asocsar
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.13.0
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.12
- PyTorch version (GPU) : 1.10.0+cu111
- Tensorflow version (GPU): 2.7.0
- Flax version: not installed
- Jax version: not installed
- JaxLib version: not installed


### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people. -->

@NielsRogge




## Information

The model used is LayoutLMv2:

The problem arises when using:
* [x] my own modified scripts:


The tasks I am working on is:
* [x] Document streaming segmentation
In my script, I try to determine when starts a new document with the objective to divide into segments a streaming of folders, so each segment can be interpreted as an independent document.

## To reproduce

Steps to reproduce the behavior:

1. Access to the colab notebook created to train LayoutLM_V2 (https://colab.research.google.com/drive/1MsEkj_WlGYDOs3vFcm1JxmMNLWj_Se78?usp=sharing)
2. Execute every cell in order
3. In the training loop, Accuracy, loss, and output will be printed, and there will be a moment when the output, Accuracy and Loss will become Nan.

## Expected behavior

The model trains, and despite if it accomplishes its task or not, the training loop ends without any Nan.


Comments:
Hi @Asocsar,

For training-related questions, can you use the [forum](https://discuss.huggingface.co/)?

I'm suspecting this has to do with the custom loss function being defined, rather than the model itself.

Thanks!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Freeze FlaxWav2Vec2 Feature Encoder
Author: sanchit-gandhi
Pull request: yes

This PR enables the FlaxWav2Vec2 Feature Encoder to be _frozen_. The flow of gradients through the Feature Encoder during forward or reverse-mode automatic differentiation is prevented through use of a simple command (`jax.lax.stop_gradient`), thus holding the Feature Encoder parameters fixed. Freezing the Feature Encoder is required for the majority of FlaxWav2Vec2 fine-tuning set-ups, making this PR an important addition to the FlaxWav2Vec2Model.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15873). All of your documentation changes will be reflected on that endpoint.

As discussed offline - the test didn't check what it was supposed to check. Bad review from my part. We should open a new PR here to fix it cc @patil-suraj (sorry should have put you as a reviewer as well)
Minor fixes for MaskFormer
Author: FrancescoSaverioZuppichini
Pull request: yes

# What does this PR do?

Fix a bug found by @Narsil  in the `feature_extractor.post_process_panoptic_segmentation`

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15916). All of your documentation changes will be reflected on that endpoint.
Fix #15898
Author: davidleonfdez
Pull request: yes

Fixes #15898

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

@sgugger


I guess it's obvious looking at the [glue example] (https://github.com/huggingface/transformers/blob/4cd7ed4b3b7360aef3a9fb16dfcc105001188717/examples/pytorch/text-classification/run_glue.py#L448) referenced in the issue, but just to be sure: is it guaranteed that the actual logits (final output of the models) are always placed in the first position?

I looked at the models of `MODEL_FOR_CAUSAL_LM_MAPPING_NAMES` and it seems a convention. I'm a bit confused because there is a `past_index` arg in `Trainer` that indexes the output of the model, which made me question if the position of the logits is guaranteed, but I guess it's never 0 or there's any other reason I don't know of. You know better than me for sure.



Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15928). All of your documentation changes will be reflected on that endpoint.

Yes, that's enforced in all the model outputs. The `past_index` is not going to temper with that :-)
Thanks a lot for fixing!
Fix FlaxDataCollatorForT5MLM for texts with "<pad>"
Author: Guitaricet
Pull request: yes

If the input text contains `"<pad>"` (true for C4/en dataset) T5 tokenizer assigns it index 0.
Collator code (line 371) assumes that only tokens with `id > 0` are the text tokens and the rest are sentinel ids (`-1`), which turns out not to be true for this particular case.
This causes the line to throw an error similar to

```
input_ids = input_ids_full[input_ids_full > 0].reshape((batch_size, -1))
*** ValueError: cannot reshape array of size 1040383 into shape (8192,newaxis)
```

This modification should also allow to use padding in T5 pre-training.

@patil-suraj (note edited by @stas00 to tag the correct maintainer for flax)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15443). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@patil-suraj , cold you please review this PR? It's very short

Oh we have a similar PR here from @patrickvonplaten

https://github.com/huggingface/transformers/pull/15835

Great!
Support CLIPTokenizerFast for CLIPProcessor
Author: cosmoquester
Pull request: yes

# What does this PR do?

Fixes #15888 

Support CLIPTokenizerFast for CLIPProcessor.
Update CLIPProcessor test code for saving and loading CLIPProcessor with fast tokenizer.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15913). All of your documentation changes will be reflected on that endpoint.
CLIPProcessor with CLIPTokenizerFast
Author: cosmoquester
Pull request: no

# 🚀 Feature request

Current `CLIPProcessor` doesn't support `CLIPTokenizerFast` requiring `CLIPTokenizer`.
In my thinking, there is no reason not to support `CLIPTokenizerFast` for `CLIPProcessor`

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

https://github.com/huggingface/transformers/blob/v4.16.2/src/transformers/models/clip/processing_clip.py#L23

it may be easy by modifying upper python code. I think I can contribute.

Comments:
Hey @cosmoquester !
The `CLIPTokenizerFast` was not used in the processor because there was an issue with it which is now fixed, cf #15067

So yes, we can now support `CLIPTokenizerFast` for `CLIPProcessor`. Feel free to open a PR!
Flax XLM-RoBERTa
Author: versae
Pull request: no
Labels: name: New model

# 🌟 New model addition: Flax XLM-RoBERTa

## Model description

The Flax version of RoBERTa and PyTorch and TF XLM-RoBERTa have been there for a while, but some pieces were missing that prevented us from doing pre-training and other tasks on Flask XLM-RoBERTa. In this PR #15900, I am adding those parts that were necessary to support XLM-RoBERTa in Flax based on the existing XLM-RoBERTa and Flax RoBERTa implementations.

## Open source status

* [x] the model implementation is available: it is based off the existing implementation of XLM-RoBERTa  and Flax RoBERTa in HuggingFace.
* [x] the model weights are available: not as Flax, but loading from PyTorch seems to work.
* [x] who are the authors: @aconneau I think, but not sure who ported it to HF. Maybe @sgugger, @patrickvonplaten?


Comments:
Add missing support for Flax XLM-RoBERTa
Author: versae
Pull request: yes

# What does this PR do?

The PyTorch and TF XLM-RoBERTa model and the Flax model for RoBERTa were added a while ago, but there were still a few missing pieces in order to be able to run pre-training and other tasks using Flax XLM-RoBERTa. This PR adds those parts.

Fixes #15910.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

@LysandreJik, @patrickvonplaten 


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15900). All of your documentation changes will be reflected on that endpoint.

Thanks a lot @versae !

I run the test using the PyTorch weights and it seems to work. I'm also already training a model using a TPUv3-8 with the code in this PR and it seems to be doing fine :) 

Thanks @versae ! Uploaded flax checkpoint for base model, and can confirm that the tests are passing. Merging!
Do not change the output from tuple to list - to match PT's version
Author: ydshieh
Pull request: yes

# What does this PR do?

Current `TFPegasusDecoder` change `all_self_attns` and `all_cross_attns` from `tuple` to `list` before return the outputs:

https://github.com/huggingface/transformers/blob/3c4fbc616f74120c3900d07c772b7d2d9c7a53dd/src/transformers/models/pegasus/modeling_tf_pegasus.py#L1061-L1065

Pytorch `PegasusDecoder` returns them as tuple (i.e. not extra step to change them to `list`). This causes (WIP) PT-TF-equivalence test failed. 

This PR fixes this, as well as for 5 other models (+ template) with the same issue:

-  [x] bart
-  [x] blenderbot
-  [x] blenderbot_small
-  [x] marian
-  [x] mbart

Think change `list` to `tuple` is acceptable (i.e. not a [too] breaking change)? cc @sgugger for this aspect.

TF: @gante @Rocketknight1 
Bart, Pegasus: @patrickvonplaten





Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15918). All of your documentation changes will be reflected on that endpoint.

I can't think of any general TF reason to prefer lists to tuples, so if the tests pass and nothing broke then LGTM!

Would like to hear from @patrickvonplaten when he has time , since he is more involved in these models.

Nice!
Tests for MaskFormerFeatureExtractor's post_process*** methods
Author: FrancescoSaverioZuppichini
Pull request: yes

# What does this PR do?
This PR adds more tests for `MaskFormerFeatureExtractor` `post_process***` methods

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15929). All of your documentation changes will be reflected on that endpoint.