noamaanMulla-03/transformers-issues huggingface dataset:
Error while finetuning XLM-R on Tensorflow-Keras
Author: Komalrathod55
Pull request: no

Hello,

I am trying to finetune XLM-RoBERTa for text classification on tensorflow-keras. I have used XLMRobertaTokenizer base to tokenize the inputs. vocab_size in config file for XLM-R model is 250002 and for Roberta it is 50265. But while training I am getting below error. Can anyone help me to solve this?

I am using google colab GPU for finetuning. Tensoflow version is 2.7.0. 

**InvalidArgumentError: indices[2,268] = 124030 is not in [0, 50265) [[node tf_roberta_for_sequence_classification_1/roberta/embeddings/Gather (defined at /usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_tf_roberta.py:149) ]] [Op:__inference_train_function_82886]

Errors may have originated from an input operation. Input Source operations connected to node tf_roberta_for_sequence_classification_1/roberta/embeddings/Gather: In[0] tf_roberta_for_sequence_classification_1/roberta/embeddings/Gather/resource:
In[1] IteratorGetNext (defined at /usr/local/lib/python3.7/dist-packages/keras/engine/training.py:866)**

Comments:
Hi @Komalrathod55 👋  Can you share the entire script (or give me a link to your colab), so I can have the entire context to check your issue? Thank you :) 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
How can I use "accelerate launch" command to run  training job on Multi-GPU?
Author: JucyCherry
Pull request: no

How can I use "accelerate launch" command to train "run_wav2vec2_pretraining_no_trainer.py" on Multi-GPU?

I found that when i use " accelerate launch run_wav2vec2_pretraining_no_trainer ..." , it would run training on a single GPU.
I also tried "accelerate launch --num_machines 4" command , but still not work

@patrickvonplaten, @anton-l

Comments:
Before starting the training, I recommend to run `accelerate config` which helps you setting up the multi-gpu training. When running the commend you are asked how many GPUs you want to use and your answer is then stored in a config that will automatically be used by `accelerate launch`. Could you give this a try?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
DebertaForMaskedLM cannot load the parameters in the MLM head
Author: yardenTal1
Pull request: no

Hello,
When I tried to use Deberta for MLM and I didn't succeed. It seems that it is a known issue.  The checkpoint provided by microsoft/deberta-base and other similar ones don't have the pre-trained weights of the Masked LM head.

https://github.com/huggingface/transformers/issues/15216

Thank you!

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
TF - update (vision_)encoder_decoder past variable
Author: gante
Pull request: yes

# What does this PR do?

I missed these 2 in the `past` refactor PR (https://github.com/huggingface/transformers/pull/15944). It reverts the error from as mentioned here (https://github.com/huggingface/transformers/pull/16230) to its previous form, as tracked here (https://github.com/huggingface/transformers/issues/15983).

Comments:
_The documentation is not available anymore as the PR was closed or merged._
Add TF benchmark examples
Author: gante
Pull request: yes

# What does this PR do?

As part of the objective to refresh TF examples, this one adds actual TF benchmarks in the examples. Also showcases the power of XLA ;)

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15853). All of your documentation changes will be reflected on that endpoint.

I see that this PR (https://github.com/huggingface/transformers/pull/15848) exists -- should I just delete the benchmark example instead?

Ah, and I didn't see that we're deprecating the benchmarks either. Maybe we shouldn't merge this!

Cc @patrickvonplaten 

@patrickvonplaten It's inconsistent with deprecating the benchmarks, so is your approval a sign we should un-deprecate them?

That's fine by me! :-)

Cool -- I'm closing the PR then and will open a new one as @patrickvonplaten suggested (moving both benchmarks to the research projects).
TF - Fix interchangeable past/past_key_values and revert output variable name in GPT2
Author: gante
Pull request: yes

# Context

From the discussion in https://github.com/huggingface/transformers/pull/16311 (PR that applies `@unpack_inputs` to TF `gpt2`): In the generate refactor, TF `gpt2` got an updated `prepare_inputs_for_generation()`, where its output `past` got renamed into `past_key_values` (i.e. as in FLAX/PT). Patrick suggested reverting it since this prepared input could be used externally.

# What did I find while working on this PR?

Reverting as suggested above makes TF `gpt2` fail tests related to `encoder_decoder`, which got an updated `prepare_inputs_for_generation()` in the same PR that expects a `past_key_values` (and not a `past`). 

Meanwhile, I've also noticed a related bug in the new `@unpack_inputs` decorator, where it was not preserving a previous behavior -- when the model received a `past_key_values` but expected a `past` input (and vice-versa), it automatically swapped the keyword. This feature was the key enabler behind `encoder_decoder`+`gpt2`, as `encoder_decoder` was throwing out `past_key_values` prepared inputs that were caught by `gpt2`'s `past` argument.

# So, what's in this PR?

This PR fixes the two issues above, which are needed for proper behavior in all combinations of inputs to TF `gpt2`, after the introduction of the decorator:
1. corrects the bug in the `@unpack_inputs` decorator and adds tests to ensure we don't regress on some key properties of our TF input handling. After this PR, `gpt2` preserves its ability to receive `past` (and `past_key_values`, if through `encoder_decoder`-like), with and without the decorator. 
2. It also reverts `past_key_values` into `past` whenever the change was introduced in https://github.com/huggingface/transformers/pull/15944, and makes the necessary changes in `encoder_decoder`-like models.

Comments:
~(Wait, there is an error)~ 
~Should be good now~ 
Nope

@patrickvonplaten hold your review, this change is not conflicting with the `encoder_decoder` models. I believe I know why, digging deeper.

_The documentation is not available anymore as the PR was closed or merged._

@patrickvonplaten now it is properly fixed -- please check the updated description at the top :)

Meanwhile, the scope increased a bit, so I'm tagging a 2nd reviewer (@Rocketknight1 )

I think Sylvain also tries to avoid `modeling_tf_utils.py` as much as possible too these days, lol. Let me take a look! 

> I'd really like to get rid of these old non-standard arguments next time we can make a breaking change, though!

@Rocketknight1 me too 🙈  that function is a mess
adds FLAUBERT to doc tests
Author: abdouaziz
Pull request: yes

# What does this PR do?

adds FLUABERT to doc tests



Comments:
Hi, @abdouaziz

Since `Flaubert` is not to be changed, and I guess you won't work on `CamemBERT` neither (right?), the changes in those 2 files have to be reverted to its original version :-) Thank you.

It also looks like you local clone and working branch doesn't have the recent update of HF `transformers`, which is necessary for this sprint. Maybe you could try to pull the latest change and rebase your working branch?
TEAMS: Training ELECTRA Augmented with Multi-word Selection
Author: stefan-it
Pull request: no
Labels: name: New model

Hi,

this [ACL paper](https://arxiv.org/abs/2106.00139) from last year (2021) proposed an ELECTRA extension:

> Pre-trained text encoders such as BERT and its variants have recently achieved state-of-the-art performances on many NLP tasks. While being effective, these pre-training methods typically demand massive computation resources. To accelerate pre-training, ELECTRA trains a discriminator that predicts whether each input token is replaced by a generator. However, this new task, as a binary classification, is less semantically informative. In this study, we present a new text encoder pre-training method that improves ELECTRA based on multi-task learning. Specifically, we train the discriminator to simultaneously detect replaced tokens and select original tokens from candidate sets. We further develop two techniques to effectively combine all pre-training tasks: (1) using attention-based networks for task-specific heads, and (2) sharing bottom layers of the generator and the discriminator. Extensive experiments on GLUE and SQuAD datasets demonstrate both the effectiveness and the efficiency of our proposed method.

Implementation is available in the TensorFlow models repository: https://github.com/tensorflow/models/tree/master/official/projects/teams

I would like to work on that, to see if it can easily be added (e.g. only writing a model conversion script).

Unfortunately, no model weights do exist at the moment. So I would like to pre-train a model and check if conversion can be done without changing the current ELECTRA implementation too much.

This issue tracks the integration into Transformers :hugs: 



Comments:
Issue that tracks generation of pre-training data: https://github.com/tensorflow/models/issues/10567
How can I use MLM and NSP to train Bert from scratch?
Author: onlyonewater
Pull request: no

How can I use MLM and NSP to train Bert from scratch?

Comments:
Hi,

I've answered this question on Stackoverflow [here](https://stackoverflow.com/questions/65646925/how-to-train-bert-from-scratch-on-a-new-domain-for-both-mlm-and-nsp/65760008#65760008).

OK, thanks!!!
"Go to definion" in VS Code doesn't work properly for Transformers
Author: Sorrow321
Pull request: no

It's not the bug of the library itself, but is it some kind of glitch that happens because of the design of the package.
The problem is when you use VS Code as an editor for your development and try to go to definion of some model, it routes you to wrong file. For example:
```
from transformers import SwinForImageClassification
model = SwinForImageClassification.from_pretrained("microsoft/swin-base-patch4-window7-224")
```
When you right click on **SwinForImageClassification** class and press "Go to Definition" it routes you to **transformers/utils/dummy_pt_objects.py** line 3634:
![image](https://user-images.githubusercontent.com/20703486/159967676-aa97230c-006b-4ecc-a55b-85784aa13cfa.png)
I don't know what this code does, but it's definetely not something that you want.
It's not specific for Swin transformer, I also experienced the same issue while working with Wav2Vec2 (I guess it's for all the models since the code is similar).

The correct file that should've been opened is **transformers/models/swin/modeling_swin.py**:
![image](https://user-images.githubusercontent.com/20703486/159968897-718b6043-d38f-4cc1-86e7-c63d448cf292.png)

I'm not sure why this happens, but maybe you should consider researching this issue and redesigning the library somehow, so that VS Code would open model code properly.

Please also let me know if you also have this problem (maybe it is just my local issue?).


Comments:
Hi @Sorrow321 -- this structure is convenient for multiple reasons, including ensuring a given import doesn't fail because you imported something that you are not planning to use, but that has a missing dependency.

I had the same problem, VS Code is a bit silly here. Go to your VS Code settings, search for `python.languageserver`, and change it to Microsoft's `Jedi` (see image below). It should solve your issue -- it did the trick for me! Let us know if it worked.
 
<img width="626" alt="Screenshot 2022-03-28 at 15 24 45" src="https://user-images.githubusercontent.com/12240844/160419773-a57f1ce4-e9f5-4c65-89d2-a27a415d2495.png">
 

Hi @gante, thank you so much, it worked.
remove references to PDF reading via PIL
Author: garfieldnate
Pull request: yes

As stated in the PIL documentation
[here](https://pillow.readthedocs.io/en/stable/handbook/image-file-formats.html?highlight=pdf#write-only-formats),
it can only write PDF's, not read them. Remove references to reading
PDF's via PIL from this page to avoid confusion.

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_15293). All of your documentation changes will be reflected on that endpoint.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Not stale, just needs a quick review and a merge.

No problem, Niels. I think the wording you suggested is still a bit confusing, but since you want to mention PDF I'll send back a counter-edit :) 

@NielsRogge I have built on your comment, making it explicit that PDFs have to be converted to images first.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Looks like I need to come back and style the file I changed.

_The documentation is not available anymore as the PR was closed or merged._

@NielsRogge I fixed the line lengths for the lines that we actually changed. The line length test is still failing, but it's because there are other lines in the file that are longer than 119 characters.
MNLI metrics overwritten 
Author: eldarkurtic
Pull request: no

## Description
When running the https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_glue.py example for the MNLI dataset, eval metrics calculated for the `validation_mismatched` dataset are overwriting the previously calculated metrics for the standard validation dataset.

## To reproduce
Simply run the default example script for text-classification with `TASK_NAME=mnli`.

## Expected behavior
When evaluation starts, the first one is done on the standard validation MNLI dataset and the metrics are saved to `eval_results.json` and `all_results.json` as:
```json
{
    "epoch": 3.0,
    "eval_accuracy": 0.8501830756712775,
    "eval_loss": 0.4692825376987457,
    "eval_runtime": 15.8047,
    "eval_samples": 9832,
    "eval_samples_per_second": 622.092,
    "eval_steps_per_second": 77.761
}
```
After that, evaluation with `validation_mismatched` happens and unfortunately it's using the same keys as the previous one (`eval_accuracy`, `eval_loss`...), which causes overwriting.
In the ideal case, it would be useful to have both metrics written in these json files.

@sgugger, @patil-suraj

Comments:
[MNLI example] Prevent overwriting matched with mismatched metrics
Author: eldarkurtic
Pull request: yes

# What does this PR do?
Fixes https://github.com/huggingface/transformers/issues/16474.

The fix appends `_mm` to the metrics for `mnli-mm` evaluation dataset to prevent overwriting eval metrics from the `mnli` dataset, and writes them together in `eval_results.json` and `all_results.json`. The MNLI-metrics look like this:

```json
{
    "eval_accuracy": 0.36067244014263883,
    "eval_accuracy_mm": 0.35506509357200977,
    "eval_loss": 1.158889889717102,
    "eval_loss_mm": 1.1670204401016235,
    "eval_runtime": 16.4691,
    "eval_runtime_mm": 15.9496,
    "eval_samples": 9815,
    "eval_samples_mm": 9832,
    "eval_samples_per_second": 595.964,
    "eval_samples_per_second_mm": 616.44,
    "eval_steps_per_second": 18.641,
    "eval_steps_per_second_mm": 19.311
}
```

@sgugger, @patil-suraj

Comments:
_The documentation is not available anymore as the PR was closed or merged._

The CI failure is unrelated to this PR (actually investigating it right now), so we can merge safely :-)
[ONNX] transformer.onnx exporting fails for longformer 
Author: RaedShabbir
Pull request: no

## Environment info

- `transformers` version: 4.16.0.dev0
- Platform: Linux-5.11.0-1024-gcp-x86_64-with-debian-bullseye-sid
- Python version: 3.7.11
- PyTorch version (GPU?): 1.10.1+cu102 (True)
- Tensorflow version (GPU?): 2.7.0 (True)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Using transformer.onnx 
- Using distributed or parallel set-up in script?: No


Models:
- Longformer: @patrickvonplaten

## Information

The same error occurs for 
- allenai/longformer-base-4096
- valhalla/longformer-base-4096-finetuned-squadv1
- allenai/longformer-large-4096

## To reproduce

Steps to reproduce the behavior:

```
python -m transformers.onnx --model=allenai/longformer-base-4096  ./longformer-base-4096 

```

### Error Code 
`RuntimeError: 0INTERNAL ASSERT FAILED at "../torch/csrc/jit/ir/alias_analysis.cpp":584, please report a bug to PyTorch. We don't have an op for aten::constant_pad_nd but it isn't a special case.  Argument types: Tensor, int[], bool, `





Comments:
Hi @RaedShabbir thank you for reporting this bug! It seems that this one slipped through our unit tests and it's not currently possible to export Longformer due to a limitation with the ops supported in JIT.

We plan to remove Longformer from the list of supported architectures for ONNX until a long-term solution can be found. In the meantime, if you'd like to investigate which part of the Longformer implementation is causing the error, then please feel free to comment here on what you find!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Working on it 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Fix missing output_attentions in PT/Flax equivalence test
Author: ydshieh
Pull request: yes

# What does this PR do?

In a previous PR #15841, `output_attentions` was not set (I accidentally removed the whole block containing it).
This PR sets  `output_attentions` to make the test more thorough.

The test still runs successfully with `1e-5` on both CPU/GPU. However, see the 2nd points in the remarks below.

It also adds `has_attentions` attribute to `FlaxModelTesterMixin` (as done in PyTorch's `ModelTesterMixin`).

# Remarks:
- In a follow up PR, we might use `has_attentions` in some existing methods (to make sure the attentions are only tested if `has_attentions` is `True`), see #15909
- There are 4 Flax model testers overwrite the Flax common `test_equivalence_pt_to_flax` and `test_equivalence_flax_to_pt`.
  - I will update them in a next PR.
  - These include `FlaxGPTJ` and `FlaxXGLM`, which will fail with `1e-5`. I need to debug them.

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_16271). All of your documentation changes will be reflected on that endpoint.

Think this (quite small) PR is ready. Nothing particular but adding the missing `config.output_attentions = self.has_attentions`.
The `super()` thing was discussed in #16280.

Will merge it today.
Fix some TF GPT-J CI testings
Author: ydshieh
Pull request: yes

# What does this PR do?

Fix some TF-GPT-J CI testing (scheduled)

- `test_mixed_precision`: require some casting
- `test_saved_model_creation` and `test_saved_model_creation_extended`: require `shape_list` instead of `shape`
- `test_model_from_pretrained`: skip for now otherwise GPU OOM

With the changes this PR, only the following test fails: `test_gptj_sample_max_time`: for example

https://github.com/huggingface/transformers/blob/c85547af2b69f9082bcd7bac97092b1d162f3fdc/tests/gptj/test_modeling_tf_gptj.py#L413

the PT gives a quite short generation sequence (say 19), while TF gives a sequence of length 256, and it takes much more time and therefore fails the tests.

I feel this remaining issue is better to be addressed in another PR.

Comments:
I need to check why

```
@unittest.skipIf(len(tf.config.list_physical_devices("GPU")) > 0, "skip testing on GPU for now to avoid GPU OOM.")
```
causes problems in other tests (torch, pipeline etc ...)


_The documentation is not available anymore as the PR was closed or merged._

Regarding the `max_time`/generate test -- when I first reviewed the GPT-J PR, I had little understanding of the current state of TF generate. Now I can tell that this test makes no sense :D Contrarily to PT's generate, TF generate has no time-based stopping criteria, so it is natural that the test fails. I'd remove it.

> Regarding the `max_time`/generate test -- when I first reviewed the GPT-J PR, I had little understanding of the current state of TF generate. Now I can tell that this test makes no sense :D Contrarily to PT's generate, TF generate has no time-based stopping criteria, so it is natural that the test fails. I'd remove it.

OK, thank you for the feedback.

But just curious (off-topic): `TF generate has no time-based stopping criteria` --> do we plan to support this in the future ??

(not very sure, but I remembered before TF can generate short sequences too. And if TF can't stop earlier, it looks like a quite big drawback ..? Anyway, we shouldn't discuss this generation thing in this PR.)

> But just curious (off-topic): `TF generate has no time-based stopping criteria` --> do we plan to support this in the future ??
> 
> (not very sure, but I remembered before TF can generate short sequences too. And if TF can't stop earlier, it looks like a quite big drawback ..? Anyway, we shouldn't discuss this generation thing in this PR.)

The plan we have for the refactoring does not mention extras like the stopping criteria, so I can only tell that it probably won't happen in the next 2-3 months :) We can generate short sentences with TF if we pass the `max_length` argument, where `generate()` generates up to `max_length` tokens.
Fix example test and test_fetcher for examples
Author: sgugger
Pull request: yes

# What does this PR do?

#16475 introduced a failure in the example tests that went undetected because the test_fetcher had the wrong path for the example test. This PR fixes both.

Comments:
Confirmed this fixes the oversight in `test_fetcher` and fixes the actual example since the example tests actually ran (and passed) here. Since this is a failure on master merging without review but happy to address comments in a follow-up PR!

_The documentation is not available anymore as the PR was closed or merged._