noamaanMulla-03/transformers-issues huggingface dataset:
About a error in retrain a xlm model
Author: zxk19981227
Pull request: no

when i train a xlm model as following code, an error occurs that 'label' is a unexpected for forward. Anyway to solve it?
`# -*- coding: UTF-8 -*-
from transformers import XLMTokenizer, XLMModel, Trainer
from datasets import load_dataset, Dataset
from transformers import LineByLineTextDataset, TrainingArguments
from transformers.data.data_collator import DataCollatorForLanguageModeling

model = XLMModel.from_pretrained('xlm-mlm-tlm-xnli15-1024')
tokenizer = XLMTokenizer.from_pretrained('xlm-mlm-tlm-xnli15-1024')
# train_datasets=load_dataset('text',data_files={'train':'./tmp/xxx.train.txt','valitation':'./tmp/all_val_data.txt'})

# 将我们刚刚加载好的datasets ，通过tokenizer做映射，得到input_id，也就是实际输入模型的东西。
# def tokenize_function(examples):
#     # Remove empty lines
#     examples["text"] = [line for line in examples["text"] if len(line) > 0 and not line.isspace()]
#     return tokenizer(
#         examples["text"],
#         padding="max_length", # 进行填充
#         truncation=True, # 进行截断
#         max_length=256, # 设置句子的长度
#         # We use this option because DataCollatorForLanguageModeling (see below) is more efficient when it
#         # receives the `special_tokens_mask`.
#         return_special_tokens_mask=True,
#     )

# 得到训练集和验证集
# train_dataset = tokenized_datasets["train"]
# eval_dataset = tokenized_datasets["validation"]
model.resize_token_embeddings(len(tokenizer))
train_dataset = LineByLineTextDataset(tokenizer=tokenizer,
                                      file_path=''
                                      block_size=512)
datacollector = DataCollatorForLanguageModeling(tokenizer, mlm=True, mlm_probability=0.15)
# train_method=Trainer(model=model,data_collator=datacollector,train_dataset=train_dataset)
training_args = TrainingArguments(output_dir='./outputs/', overwrite_output_dir=True, num_train_epochs=20,
                                  learning_rate=6e-5,
                                  per_device_train_batch_size=128, save_total_limit=10)  # save_steps=10000
trainer = Trainer(
    model=model, args=training_args, data_collator=datacollector, train_dataset=train_dataset)


trainer.train()
trainer.save_model('./outputs/')
`

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Tracking variables other than loss during training
Author: umgupta
Pull request: no

# 🚀 Feature request

Allow to track other variables during training with the [trainer](https://huggingface.co/transformers/main_classes/trainer.html).

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation
Often during training one wants to track variables other than just the loss. For example, the loss may be consisting of two different components and the user may want to track the two separately. As of now, the trainer can only track loss. It would be great if a user could simply pass the list of keys of auxiliary losses that they may want to track.   
<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution
I am happy to discuss and contribute code for this. 
<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


@sgugger 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
conversion wav2vec2 model from fairseq to huggingface
Author: shiva1393
Pull request: no
Labels: name: Migration

# 📚 Migration
Hi, 
trained wav2vec2 model in fairseq for my own dataset. Now i need to finetune pretrained fairseq wav2vec2 model. To train hugging face model it is asking following files.
1. config.json
2. preprocessor_config.json
3. pytorch_model.bin
4. special_tokens_map.json
5. tokenizer_config.json
6. vocab.json

To get this files i used following command for wav2vec2 base model
 cp transformers/src/transformers/models/wav2vec2/convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py .
 wget https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_small_960h.pt -O ./wav2vec_small_960h.pt
 mkdir dict
 wget https://dl.fbaipublicfiles.com/fairseq/wav2vec/dict.ltr.txt
 mkdir outputs
 python convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py --pytorch_dump_folder_path ./outputs --checkpoint_path ./wav2vec_small_960h.pt --dict_path dict.ltr.txt

This following commands are working fine for wav2vec_small_960h.pt model and able to generate config.json, preprocessor_config.json, pytorch_model.bin, special_tokens_map.json,  tokenizer_config.json, vocab.json 

But i trained model using following command

fairseq-hydra-train \
    task.data=/path/to/data \
    --config-dir /path/to/fairseq-py/examples/wav2vec/config/pretraining \
    --config-name wav2vec2_base_librispeech
 i got checkpoint_mine.pt

Then i  used 

 python convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py --pytorch_dump_folder_path ./outputs --checkpoint_path ./checkpoint_mine.pt --dict_path dict.ltr.txt

I'm getting following error
File "convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py", line 255, in <module>
    convert_wav2vec2_checkpoint(
  File "env/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py", line 240, in convert_wav2vec2_checkpoint
    recursively_load_weights(model, hf_wav2vec, not is_finetuned)
  File "convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py", line 134, in recursively_load_weights
    set_recursively(hf_model, mapped_key, value, name, weight_type)
  File "convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py", line 71, in set_recursively
    hf_pointer = getattr(hf_pointer, attribute)
  File "env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 947, in __getattr__
    raise AttributeError("'{}' object has no attribute '{}'".format(

Please can any one suggest what i need to change 
why  wav2vec_small_960h.pt  and checkpoint_mine.pt are differ. I saw previous discuusions but i didn't get proper solution.
Thanks in advance

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
T5: Create position related tensors directly on device instead of CPU
Author: armancohan
Pull request: yes

# What does this PR do?

The current implementation of the `compute_bias` function of the T5 model creates tensors on the CPU (`memory_position` and `context_position`) and then moves them to the corresponding device with `.to()`. 
While this has minimal impact in single-gpu training, in multi-gpu large batch training, as the number of GPUs increases this reduces GPU utilization.
This short PR addresses the issue. 

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR (@patrickvonplaten).


Comments:
Skip ProphetNet test
Author: LysandreJik
Pull request: yes

cc @patrickvonplaten 

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Not getting the same results with run_qa and run_qa_no_trainer scripts
Author: eldarkurtic
Pull request: no

## Environment info
Just followed the default setup instructions in a new conda environment:
```shell
git clone https://github.com/huggingface/transformers
cd transformers
pip install .
pip install -r examples/pytorch/question_answering/requirements.txt
```

### Who can help
@sgugger @patil-suraj 

## Information
Model I am using: https://huggingface.co/prajjwal1/bert-tiny

My goal is to run the finetuning example on the bert-tiny model and Squad dataset, with and without the Trainer class, and to obtain the same results.

## The problem
Running with the Trainer class with:
```shell
CUDA_VISIBLE_DEVICES=0 python run_qa.py \
  --model_name_or_path prajjwal1/bert-tiny \
  --dataset_name squad \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /trainer_squad/ 
```
ends with: `eval_exact_match = 31.3245` and `eval_f1 = 43.3655`.

Then, running the same setup but without the Trainer with:
```shell
CUDA_VISIBLE_DEVICES=0 python run_qa_no_trainer.py \
  --model_name_or_path prajjwal1/bert-tiny \
  --dataset_name squad \
  --per_device_train_batch_size 12 \
  --learning_rate 3e-5 \
  --num_train_epochs 2 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir /no_trainer_squad/
```
ends with: `Evaluation metrics: {'exact_match': 19.981078524124882, 'f1': 32.57782310536579}`.

It's interesting that I obtain the same results (`F1 = 49.73` and `EM = 48.6`) when I run with and without Trainer class, but with a different dataset: `--dataset_name squad_v2` and `--version_2_with_negative`.

Comments:
The two scripts are different, they do not have the same defaults for the `seed` and even then, they do not randomize the data the same way. It's impossible for the two of them to give you the same results. 

Do you have any suggestions on what to modify in the `run_qa_no_trainer.py` to get a bit better results (possibly closer to "the baseline" with `run_qa.py`)? 
Based on the same results I got with the `squad_v2`, I assumed that both scripts are doing the same thing, and that the one without the Trainer just provides more flexibility to modify stuff in the train/eval loops. 

On one GPU, you could maybe achieve the same results by passing the same seeds, making sure it's set at the right place, but that's a big maybe. Reproducibility is a hard enough problem with one script, so with two scripts that start with different assumptions and use different APIs, it's next to impossible.

Okay, got it! I then just misunderstood their roles in the examples project. Thanks for a quick reply.
GPT-Neo ONNX export
Author: michaelbenayoun
Pull request: yes

# What does this PR do?

This PR enables the export of GPT-Neo to ONNX by extending the new module transformers.onnx.

It also provides a possible way of implementing the export for specific tasks: the task can be specified when instantiating an OnnxConfig. It is a nice approach because it makes factoring most of the code for the inputs / outputs very easy, but it is less aligned with transformers DNA than having subclasses (such as OnnxConfigForSequenceClassification, etc)  taking care of that.

The issue with having many subclasses is that it would have to be done everytime one wants to add the support for a model. 
What do you think?

Comments:
@sgugger @LysandreJik What do you think would be the best way to approach this exporting features for downstream task? I think we have the two possible ways:

- One config per task `XOnnxConfigForY` => Follow the general "duplication" pattern in transformers
- One config with task as parameter encapsulating the logic for I/O for each possible task => Potentially reduce the LoC

I think using a `task` argument is a nice way of avoiding too many new classes which would crowd the main init of transformers.

@michaelbenayoun is the PR ready for review? 🥰 

> @michaelbenayoun is the PR ready for review?
Yes, it is!

I also implemented a "factory"  called `FeaturesManager` located  at `onnx/features.py` from what was done before by @mfuntowicz in `onnx/__main__.py` which manages the mapping between features and models / onnx configs.

From what @sgugger [said](https://github.com/huggingface/transformers/pull/12911#issuecomment-887502770), I went with the "task argument" approach. Basically, a feature is the combination of a task and the potential use of past keys and values, for instance:

- sequence-classification
- sequence-classification-with-past

Any feature containing "-with-past" will be mapped by the factory to an OnnxConfig instantiated using the `with_past` method.

@mfuntowicz any comments on the changes I have made?
Set dropout for ClassificationHead
Author: PhilipMay
Pull request: no

When I train a classification task I want to be able to set a dropout value for the prediction head.
This dropout value should be independent of the other dropout values of the language model.

Now the issue is: If I want to set a dropout for the ClassificationHead of Electra (and other models) I have to set the 
`hidden_dropout_prob`. See here:

https://github.com/huggingface/transformers/blob/546dc24e0883e5e9f5eb06ec8060e3e6ccc5f6d7/src/transformers/models/electra/modeling_electra.py#L903

But this sets (changes) the dropout of many other layers too. IMO that should not be the case.

IMO there should be an dedicated dropout for the head which is defaulted by `hidden_dropout_prob` but can be changed by
the user.

@sgugger and @LysandreJik what do you think?

PS: I can provide a PR if wanted...

Comments:
Valid point, some models (like mBART, DETR, PEGASUS) have a `classifier_dropout` in their config. 

It should ideally be the case for all models.

Agreed with Niels. If you want to make a PR @PhilipMay, please go ahead. Just make sure to use the same attribute name ( `classifier_dropout`) as in the existing models, and make sure there is no breaking change (which should be the case with your proposition).

> and make sure there is no breaking change (which should be the case with your proposition).

@sgugger I am confused - sorry.
You mean my suggestion:

> IMO there should be an dedicated dropout for the head which is defaulted by hidden_dropout_prob but can be changed by
the user.

Adds no breaking change and is ok? Or do you say it adds a breaking change?

Well - I just started with Electra: #12794

@sgugger can you please check that if it follows the right pattern? If yes I can add the other models.

This is closed by the PR merge.
`fill-mask` pipeline provides `<mask>` token among predictions
Author: rspreafico-absci
Pull request: no

## Environment info
- `transformers` version: 4.8.1
- Platform: Linux-5.4.0-74-generic-x86_64-with-glibc2.31
- Python version: 3.9.5
- PyTorch version (GPU?): 1.9.0+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: see below
- Using distributed or parallel set-up in script?: no

### Who can help
@LysandreJik
@sgugger

## Information

Model I am using (Bert, XLNet ...): RoBERTa

The problem arises when using:
* [X] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

```
from transformers import RobertaTokenizerFast, RobertaForMaskedLM, pipeline

tokenizer = RobertaTokenizerFast.from_pretrained("roberta-base")
model = RobertaForMaskedLM.from_pretrained("roberta-base")

fill_mask = pipeline(
    "fill-mask",
    model=model,
    tokenizer=tokenizer,
)

resp = fill_mask("My <mask> is Roberto.", top_k=len(tokenizer.get_vocab()))

[x for x in resp if x['token'] == tokenizer.mask_token_id]
```

## Expected behavior

Because the job of the `fill-mask` pipeline is to fill the `<mask>` special token, the expectation is that `<mask>` itself is not part of the possible predictions.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[Flax] Correct pt to flax conversion if from base to head
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Currently loading a base model into a head model while using `from_pt=True` is broken in Flax. 
E.g. the following fails:

```python
from transformers import RobertaModel, FlaxRobertaForMaskedLM, RobertaConfig

model = RobertaModel(RobertaConfig())
model.save_pretrained("./")

FlaxRobertaForMaskedLM.from_pretrained("./", from_pt=True)
```
It's not that trivial to correct it, since the conversion PT => Flax requires some renaming which is a bit "hacky". To solve the problem this PR now always checks whether both the weight name with and withouth base model prefix is expected. If one of the is expected -> then the weight name is correctly changed. 

Tests are added to ensure that all models will be correctly converted from PyTorch in the future.


Comments:
[Flax T5] Speed up t5 training
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

This PR makes sure that no jax functionality is used in the preprocessing to make sure the TPU is not unnecessarly blocked. This small change leads to a **5x** factor speed-up in training T5.

🚨🚨 **Note**: It is extremely important to verify that no DeviceArrays are created during the data preprocessing to make sure that the training step can run asynchronously on TPU while the preprocessing runs on CPU. A good rule is to make sure that in the training step look, only the function `p_train_step` uses JAX/Flax code and all other functions run on CPU. Other relevant links: https://jax.readthedocs.io/en/latest/async_dispatch.html#async-dispatch 🚨🚨



Comments:
The PR is tested here: https://huggingface.co/patrickvonplaten/t5-base-norwegian/tensorboard (check train loss graph which shows that time is reduced to < 5h now)
FX submodule naming fix
Author: michaelbenayoun
Pull request: yes

This PR is related to HFTracer, the class responsible for allowing torch.fx symbolic tracing on transformers models.

It enhances the way dynamically inserted modules are named, making the name of the submodule inserted to the parent more explicit and close to what the submodule represents. It also solves issues related to the way submodule were inserted: instead of using `setattr`, `nn.Module.add_module()` is used.

Comments:
Reference postprocess_qa_predictions score method
Author: LozanoAlvarezb
Pull request: no

To the best of my knowledge literature on question answering computes the probability of a span by first computing the probability for a token to be the start token or the end token independently and then multiplying those probabilities. In contrast, the [postprocess_qa_predictions](https://github.com/huggingface/transformers/blob/57461ac0b4e4f7349c2437fcf8d4115014d6ceda/examples/pytorch/question-answering/utils_qa.py#L31) function computes the probability for an answer by first summing both logits, ranking them, and applying the softmax over the top n_best scores.

Is there any reference in the literature supporting this?



Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
(WIP) Add FNet with flax template
Author: changjonathanc
Pull request: yes

# What does this PR do?

This is a demo for #12441

Fixes #12411 


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.