noamaanMulla-03/transformers-issues huggingface dataset:
Cant load google/reformer-enwik8
Author: hadaev8
Pull request: no


OSError: Can't load tokenizer for 'google/reformer-enwik8'. Make sure that:

- 'google/reformer-enwik8' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'google/reformer-enwik8' is the correct path to a directory containing relevant tokenizer files

https://huggingface.co/google/reformer-enwik8?text=My+name+is+Julien+and+I+like+to

Comments:
Duplicate of #11649
Add visual + link to Premium Support webpage
Author: julien-c
Pull request: yes

Close #11635

Comments:
Built docs is at https://212403-155220641-gh.circle-artifacts.com/0/docs/_build/html/index.html

<img width="1360" alt="Screenshot 2021-05-17 at 11 08 58" src="https://user-images.githubusercontent.com/326577/118463639-ff238180-b6cd-11eb-8c45-a6a1e6471b60.png">


CI failure seems unrelated
ValueError: could not broadcast input array from shape (2816,384) into shape (2698,384)
Author: zhijxu-MS
Pull request: no

the command to reproduce:
cd huggingface-transformers/examples/pytorch/question-answering
python -m torch.distributed.launch --nproc_per_node=8 ./run_qa.py \
		  --model_name_or_path roberta-large \
		  --dataset_name squad  \
		  --do_train --do_eval \
		  --per_device_train_batch_size 16 \
		  --per_device_eval_batch_size 256 \
		  --learning_rate 3e-5 \
		  --num_train_epochs 2 \
		  --max_seq_length 384 \
		  --doc_stride 128 \
		  --output_dir test_result2/$trials --overwrite_output_dir \
		  --logging_dir test_result2/$trials/tensorboard --logging_first_step --logging_steps 50 \
                  --fp16



i tried add "--max_eval_samples 10240", this will fix the error, while the AUC result is quite low(exact_match  = 4.9414, f1 = 8.9784). and when i ran with 1gpu, the above command can succeed(exact_match  = 88.5336, f1 = 94.3266)


the full error is "File "./transformers/src/transformers/trainer_pt_utils.py", line 410, in _nested_set_tensors
    i * slice_len : (i + 1) * slice_len
    i * slice_len : (i + 1) * slice_len
ValueError: could not broadcast input array from shape (2816,384) into shape (2698,384)"

Comments:
@sgugger i saw you have a PR to fix similar error, could you help to take a look?



You should upgrade your version of Transformers: this code is not used anymore inside the `Trainer`.

i am installing transformers from source, the issue still threr.

and issue throw from https://github.com/huggingface/transformers/blob/86d5fb0b360e68de46d40265e7c707fe68c8015b/src/transformers/trainer_pt_utils.py#L411, looks like the code is still used in master?



Indeed, the subclass of the `Trainer` for QA still uses the old code! The PR linked above should fix this.
Thanks for flagging!
error while training wave2vec on arabic text
Author: omerarshad
Pull request: no

Traceback (most recent call last):
  File "/content/transformers/examples/research_projects/wav2vec2/run_asr.py", line 480, in <module>
    main()
  File "/content/transformers/examples/research_projects/wav2vec2/run_asr.py", line 430, in main
    num_proc=2,
  File "/usr/local/lib/python3.7/dist-packages/datasets/dataset_dict.py", line 448, in map
    for k, dataset in self.items()
  File "/usr/local/lib/python3.7/dist-packages/datasets/dataset_dict.py", line 448, in <dictcomp>
    for k, dataset in self.items()
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py", line 1289, in map
    update_data = does_function_return_dict(test_inputs, test_indices)
  File "/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py", line 1260, in does_function_return_dict
    function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "/content/transformers/examples/research_projects/wav2vec2/run_asr.py", line 423, in prepare_dataset
    batch["labels"] = processor(batch[data_args.target_text_column]).input_ids
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/wav2vec2/processing_wav2vec2.py", line 117, in __call__
    return self.current_processor(*args, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py", line 2266, in __call__
    **kwargs,
  File "/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py", line 2451, in batch_encode_plus
    **kwargs,
  File "/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils.py", line 543, in _batch_encode_plus
    verbose=verbose,
  File "/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils.py", line 606, in _batch_prepare_for_model
    return_attention_mask=return_attention_mask,
  File "/usr/local/lib/python3.7/dist-packages/transformers/tokenization_utils_base.py", line 2579, in pad
    while len(required_input[index]) == 0:
IndexError: list index out of range
[ ]


Comments:
Hello, Can you provide more information about the dataset you are using. The error message alone seems very vague, but pointing towards input.

Its my custom dataset. Arabic audio and arabic transcription

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
ZeroDivisionError: float division by zero after some epochs while training using run_mmimdb.py
Author: suyogkute
Pull request: no

I am trying to train image+test model using the run_mmiddb.py(https://github.com/huggingface/transformers/tree/master/examples/research_projects/mm-imdb). I have two classes in this task.
Initially i ran it for 1 epoch with below input parameters and it went well::
--model_name_or_path bert-base-cased \
--max_seq_length 512 \
--stride_len 112 \
--num_image_embeds 3 \
--per_gpu_train_batch_size 8 \
--per_gpu_eval_batch_size 16 \
--gradient_accumulation_steps 20 \
--patience 5 \
--fp16 \

and from log i have "Num examples = 5703" while training and in evaluation "Num examples = 1176".

Now, when i am running it for 20 epochs and reduced gradient_accumulation_steps to 6. It gave error as
"ZeroDivisionError: float division by zero"

maybe after 13 epochs i guess(I lost log some of it).
Also the loss from startig onwards is "nan". I dont know how. I see the loss fucntion as 
"criterion = nn.BCEWithLogitsLoss(pos_weight=label_weights)"
Even if loss is "nan" from starting, I see 'macro_f1' going up from 55 to 58 and again back to 55 before this above mentioned final error.
Any suggestion/solution is appreciated. Thanks

Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Why optimizer need split parameter group?
Author: MarsSu0618
Pull request: no

I want to ask some question about follow codes:
```
no_decay = ['bias', 'LayerNorm.weight']
optimizer_grouped_parameters = [
    {'params': [p for n, p in model.named_parameters() if not any(nd in n for nd in no_decay)], 'weight_decay': 0.01},
    {'params': [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
]
optimizer = AdamW(optimizer_grouped_parameters, lr=1e-5)
```

if i hide `{'params': [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}`.
Would it cause other problems? and why need split parameters group.

reference: https://huggingface.co/transformers/training.html#fine-tuning-in-native-pytorch

Comments:
Difficult to tell what happens when you apply weight_decay to those layers as well. I think you should just give it a try and tells us what has happened. 
The shown code that applies different weight_decay to different parameters is in line with the original bert implementation ([link](https://github.com/google-research/bert/blob/eedf5716ce1268e56f0a50264a88cafad334ac61/optimization.py#L65)).

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
Using run_language_modeling.py to train an English adapter
Author: JackyXiangcheng
Pull request: no
Labels: name: Migration

# 📚 Migration

## Information

<!-- Important information -->

Model I am using bert-base-multilingual-cased:

Language I am using the model on English:

The problem arises when using:

When I entered the codes in the command line and run, the process just stuck here for a long time and did nothing, I tried many times and it always couldn’t start to train?

![image](https://user-images.githubusercontent.com/40454951/114124128-4af04780-9926-11eb-8241-2ed162171a72.png)



The tasks I am working on is:
* Train an English adapter using this script: https://github.com/Adapter-Hub/adapter-transformers/blob/master/examples/contrib/legacy/run_language_modeling.py
* I wrote this in command line:
* python3 run_language_modeling.py \
--output_dir=xxx \
--model_type=bert \
--model_name_or_path=bert-base-multilingual-cased \
--do_train \
--train_data_file=xxx/a.txt \
--do_eval \
--eval_data_file=xxx/b.txt \
--mlm \
--language en \
--train_adapter \
--adapter_config pfeiffer \
--per_gpu_train_batch_size 4 \
--per_gpu_eval_batch_size 4 \
--learning_rate 5e-5

## Details

<!-- A clear and concise description of the migration issue.
    If you have code snippets, please provide it here as well.
    Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
    Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.
    -->

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
 
- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

<!-- IMPORTANT: which version of the former library do you use? -->
* `pytorch-transformers` or `pytorch-pretrained-bert` version (or branch):


## Checklist

- [ ] I have read the migration guide in the readme.
 ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers);
  [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [ ] I checked if a related official extension example runs on my machine.


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
cannot import name 'BigBirdModel' from 'transformers'
Author: Shengyu-Liu558
Pull request: no

when I write the “from transformers import BigBirdModel”， the error is “cannot import name 'BigBirdConfig' from 'transformers'”,how to solve the problem? thank you. 

Comments:
Hello! Please respect the issue template so that we can help you. Big Bird is only available in the latest transformers version, do you have this version in your setup?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
save_strategy="no" but checkpoints are created after each evaulation
Author: hassanzadeh
Pull request: no

I have set the save_strategy to no in my TrainingArguments, yet, checkpoints are being created every 100 steps, here is how I configured TA:
<img width="790" alt="image" src="https://user-images.githubusercontent.com/13952413/113585244-faad7880-95f9-11eb-90f5-069a1a688723.png">



Comments:
Is that a bug or something is wrong with my config?


I wonder if that could be because of load_best_model_at_end? In that case, how can I remove all remaining checkpoints after training is done?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
[Documentation issue] Sequence to sequence models
Author: LysandreJik
Pull request: no

Most models have their example docstrings appended using the `add_code_sample_docstrings` method. This method checks the name of the architecture, and according to its suffix, adds the corresponding sequence.

However, this isn't perfect with the difference between sequence-to-sequence and enc/dec only models, which can have the same suffix: `BertModel` and `MarianModel` have the same suffixes, but should not have the same docstrings, the latter needing `decoder_input_ids` in order to work.

I propose to return a different code sample according to the architecture type (seq-to-seq vs enc/dec). The signature of the method that updates the code sample is the following:

https://github.com/huggingface/transformers/blob/2d458b2c7d6fb1dd5b2361938d1b5bd4c2106479/src/transformers/file_utils.py#L884-L886

There are in my opinion two ways to go about it:

- We can infer the type from the `output_type`. We've mentioned some while ago that having inheritance with model outputs and being able to identify categories of outputs from their classes would make sense for a potential pipeline-v2 implementation, this could be implemented here to detect if a model is seq-2-seq or not.
- We could add another argument to the signature too mention if it's seq-2-seq or not. However, this isn't very future-proof and can result in increased complexity if we ever need an additional separation across architectures.

@patrickvonplaten @sgugger @patil-suraj looking forward to your feedback. I tend to prefer the first option even if it requires a bit more work.

Related issue: https://github.com/huggingface/transformers/issues/10368

Comments:
Option 1) seems the way to go for me as well here!

Said the same in private but realized I didn't put it here. So option 1 it is!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Don't close this one robot!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
ELECTRA-large-discriminator results are not stable
Author: ngoquanghuy99
Pull request: no

I'm fine tuning ELECTRA-large-discriminator (https://huggingface.co/google/electra-large-discriminator) for my classification task.
The problem is that the results are not stable. First time I fine tuned it with validation accuracy around 97%. The second try, i got 7x% accuracy. The third is 5x% accuracy. 
I just did the same job with BERT, RoBERTa,...
Anyone has any thoughts on this problem?


Comments:
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
finetune_trainer with python -m torch.distributed.launch
Author: rabeehkarimimahabadi
Pull request: no

Hi
I need to run the finetune_trainer with multiple gpus, I am getting the error of 

"Default process group is not initialized"
AssertionError: Default process group is not initialized

I am using custom dataloader, might be hard to share all the parts of codes, but I defined sampler as DistributedSampler. 
this is transformer 3.5.1, python 3.7 on GPU.

thanks
Best
Rabeeh 



Comments:
Also, could you add the command to run distributed training with GPUs with finetune_trainer in README? thanks 

I tried with latest version of transformers on 4 gpu with distributed training 

rabeeh@gpu4:~/transformers/examples/seq2seq$ python -m torch.distributed.launch finetune.py  --learning_rate=3e-5     --fp16     --gpus 4     --do_train     --do_predict     --n_val 1000     --val_check_interval 0.1     --data_dir wmt_en_ro     --train_batch_size=1     --eval_batch_size=1     --output_dir=xsum_results     --num_train_epochs 1     --model_name_or_path t5-smal

getting the following error, thanks 

finetune.py: error: unrecognized arguments: --local_rank=0
Traceback (most recent call last):
  File "/opt/conda/envs/transformers/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/opt/conda/envs/transformers/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/transformers/lib/python3.7/site-packages/torch/distributed/launch.py", line 260, in <module>
    main()
  File "/opt/conda/envs/transformers/lib/python3.7/site-packages/torch/distributed/launch.py", line 256, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/conda/envs/transformers/bin/python', '-u', 'finetune.py', '--local_rank=0', '--learning_rate=3e-5', '--fp16', '--gpus', '4', '--do_train', '--do_predict', '--n_val', '1000', '--val_check_interval', '0.1', '--data_dir', 'wmt_en_ro', '--train_batch_size=1', '--eval_batch_size=1', '--output_dir=xsum_results', '--num_train_epochs', '1', '--model_name_or_path', 't(trans(tr(transformers) 



This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

@rabeehk I also encountered the same problem as you. Have you solved it?
Fix checkpoint deletion
Author: sgugger
Pull request: yes

# What does this PR do?

As pointed out on the [forums](https://discuss.huggingface.co/t/checkpoint-missing-optimizer-pt-how-to-resume/6138) there is a problem in the way checkpoints are deleted currently when `save_total_limit` is set and `load_best_model_at_end` is True. Since the best checkpoint is switched with the last checkpoint, we end up deleting the last checkpoint instead of the oldest available one.

This PR fixes this issue and adds tests.

Comments:
Add more subsections to main doc
Author: patrickvonplaten
Pull request: yes

# What does this PR do?

This PR adds a subsection right before the list of supported models & the big table of supported frameworks for each model. Merging this PR would change the "welcome" doc page as follows:

![pic1](https://user-images.githubusercontent.com/23423619/118638456-c4edd900-b7ce-11eb-8191-aa9bfc62248a.png)

and 

![pic2](https://user-images.githubusercontent.com/23423619/118638477-ca4b2380-b7ce-11eb-8dbe-f9d68b3c6908.png)

The motivation for this PR is mainly to be able to better link to all supported models and framework. *E.g.* when asking which models are supported by Flax, it's nice to have a direct link instead of having to scroll down 

Comments:
Rename `nlp` variables into more appropriate names
Author: Narsil
Pull request: no
Labels: name: Good First Issue

# 🚀 Feature request

In `pipelines` tests and documentation they are recurringly named `nlp`, the goal is to rename them to something more appropriate.

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

## Motivation


```python
nlp = pipeline(task='conversational', model='XXX')
```
This is a bit pretentious as it does cover all NLP, and a better name help understanding to users too.
For instance the `conversational` task pipeline could be named `conversational_agent`. Or maybe still a generic name but less
pretentious `pipe`, `pipeline` (caveat: those are less clear in what they intend to achieve)

The goal is to rename all occurences of `nlp` by better names within both tests and the documentation.


<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

## Your contribution

The better names could be discussed here.

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->


Comments:
Thanks for creating the issue! Here are examples of what good names are in my humble opinion:
```
classifier = pipeline("sentiment-analysis")
unmasker = pipeline("fill-mask")
text_generator = pipeline("text-generation")
```
In short, something that describes the task it achieves.

Hi guys, I'm new and I'd like to start helping out. Can I take over this request? And to clarify, you're referring to the references primarily in the /tests directory and in the /docs directory?

Hi @terrenceedmonds , Thanks for taking this on ! 

Yes, for both directories, but docs are also found within docstrings within the `src/transformers/pipelines` directory.

Hii. Is this issue still open? I want to take this issue. Also, this will be my first contribution. Any help in getting me started will be highly appreciated.

Let's see if @terrenceedmonds wants to to finish it first (the PR was almost ready to merge).

Can I work on this issue, if @terrenceedmonds is not working on it?

Yes, you can go ahead!