Longformer training : CUDA error: device-side assert triggered

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:
- Python version: 3.7
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: sharedddp (Fairscale)

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten

Library:
- trainer: @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): Longformer

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ x ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ x ] my own task or dataset: (give details below)

## To reproduce

When i use the same configuration to train model type bert it works but this does not work for longformer.
Steps to reproduce the behavior:
/opt/conda/bin/python -m torch.distributed.launch \
--nnodes=$WORLD_SIZE \
--node_rank=$RANK \
--master_addr=$MASTER_ADDR \
--master_port=$MASTER_PORT \
--nproc_per_node=1 $SCRIPT \
--output_dir=$OUT_DIR \
--logging_dir=$OUT_DIR \
--tokenizer_name=$TOKENIZER \
--model_type=longformer --do_train --do_eval \
--cache_dir=$CACHE_DIR \
--overwrite_cache \
--validation_file=$EVAL_DATA \
--overwrite_output_dir \
--train_file=$TRAIN_DATA_FOLDER \
--dataset_name=$DATASET_NAME \
--line_by_line \
--learning_rate=${INIT_LR} \
--save_steps=${SAVE_STEPS} \
--max_seq_length=${BLOCK_SIZE} \
--gradient_accumulation_steps=${GRAD_ACCUM_STEPS} \
--fp16 \
--num_train_epochs=$EPOCHS \
--per_device_train_batch_size=$BATCH_SIZE_PER_GPU \
--local_rank=$LOCAL_RANK \
--train_dataset_info_path=$TRAIN_DATASET_INFO \
--test_dataset_info_path=$TEST_DATASET_INFO \
--sharded_ddp \

Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    train_result = trainer.train(resume_from_checkpoint=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in train
    tr_loss += self.training_step(model, inputs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1443, in training_step
    loss = self.compute_loss(model, inputs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1477, in compute_loss
    outputs = model(**inputs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/fairscale/nn/data_parallel/sharded_ddp.py", line 218, in forward
    return self.module(*inputs, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/models/longformer/modeling_longformer.py", line 1765, in forward
    return_dict=return_dict,
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/models/longformer/modeling_longformer.py", line 1669, in forward
    return_dict=return_dict,
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/models/longformer/modeling_longformer.py", line 1245, in forward
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
    is_global_attn = is_index_global_attn.flatten().any().item()
RuntimeError: CUDA error: device-side assert triggered
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    train_result = trainer.train(resume_from_checkpoint=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in train
    train_result = trainer.train(resume_from_checkpoint=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in train
    tr_loss += self.training_step(model, inputs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1443, in training_step
    tr_loss += self.training_step(model, inputs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1443, in training_step
    loss = self.compute_loss(model, inputs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1477, in compute_loss
    loss = self.compute_loss(model, inputs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1477, in compute_loss
    outputs = model(**inputs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    outputs = model(**inputs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/fairscale/nn/data_parallel/sharded_ddp.py", line 218, in forward
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/fairscale/nn/data_parallel/sharded_ddp.py", line 218, in forward
    return self.module(*inputs, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    return self.module(*inputs, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/models/longformer/modeling_longformer.py", line 1765, in forward
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/models/longformer/modeling_longformer.py", line 1765, in forward
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
    return_dict=return_dict,
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    return_dict=return_dict,
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/models/longformer/modeling_longformer.py", line 1669, in forward
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/models/longformer/modeling_longformer.py", line 1669, in forward
Traceback (most recent call last):
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 661, in <module>
    return_dict=return_dict,
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    return_dict=return_dict,
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/models/longformer/modeling_longformer.py", line 1245, in forward
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/models/longformer/modeling_longformer.py", line 1245, in forward
    is_global_attn = is_index_global_attn.flatten().any().item()
RuntimeError: CUDA error: device-side assert triggered
    is_global_attn = is_index_global_attn.flatten().any().item()
RuntimeError: CUDA error: device-side assert triggered
    train_result = trainer.train(resume_from_checkpoint=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in train
    main()
  File "/data/atc_tenant/bert_data/smancha5/run_mlm.py", line 465, in main
    tr_loss += self.training_step(model, inputs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1443, in training_step
    loss = self.compute_loss(model, inputs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1477, in compute_loss
    train_result = trainer.train(resume_from_checkpoint=model_path)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1003, in train
    outputs = model(**inputs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    tr_loss += self.training_step(model, inputs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1443, in training_step
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/fairscale/nn/data_parallel/sharded_ddp.py", line 218, in forward
    return self.module(*inputs, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/models/longformer/modeling_longformer.py", line 1765, in forward
    loss = self.compute_loss(model, inputs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/trainer.py", line 1477, in compute_loss
    return_dict=return_dict,
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    outputs = model(**inputs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/models/longformer/modeling_longformer.py", line 1669, in forward
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/fairscale/nn/data_parallel/sharded_ddp.py", line 218, in forward
    return self.module(*inputs, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/models/longformer/modeling_longformer.py", line 1765, in forward
    return_dict=return_dict,
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/models/longformer/modeling_longformer.py", line 1245, in forward
    return_dict=return_dict,
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    is_global_attn = is_index_global_attn.flatten().any().item()
RuntimeError: CUDA error: device-side assert triggered
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/models/longformer/modeling_longformer.py", line 1669, in forward
    return_dict=return_dict,
  File "/opt/conda/lib/python3.6/site-packages/torch/nn/modules/module.py", line 726, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python3.6/site-packages/transformers/models/longformer/modeling_longformer.py", line 1245, in forward
    is_global_attn = is_index_global_attn.flatten().any().item()
RuntimeError: CUDA error: device-side assert triggered
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: device-side assert triggered
Exception raised from create_event_internal at ../c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fc78c43d99b in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xc10 (0x7fc78c680280 in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7fc78c425dfd in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #3: <unknown function> + 0x5414e2 (0x7fc7c549d4e2 in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #4: <unknown function> + 0x19aaae (0x5603f8975aae in /opt/conda/bin/python)
frame #5: <unknown function> + 0xf2868 (0x5603f88cd868 in /opt/conda/bin/python)
frame #6: <unknown function> + 0x1f0d91 (0x5603f89cbd91 in /opt/conda/bin/python)
frame #7: <unknown function> + 0xf270d (0x5603f88cd70d in /opt/conda/bin/python)
frame #8: <unknown function> + 0x19aa90 (0x5603f8975a90 in /opt/conda/bin/python)
frame #9: <unknown function> + 0xf2868 (0x5603f88cd868 in /opt/conda/bin/python)
frame #10: <unknown function> + 0x1f0d91 (0x5603f89cbd91 in /opt/conda/bin/python)
frame #11: <unknown function> + 0xf2828 (0x5603f88cd828 in /opt/conda/bin/python)
frame #12: <unknown function> + 0x19aa90 (0x5603f8975a90 in /opt/conda/bin/python)
frame #13: <unknown function> + 0xf2868 (0x5603f88cd868 in /opt/conda/bin/python)
frame #14: <unknown function> + 0x1f0d91 (0x5603f89cbd91 in /opt/conda/bin/python)
frame #15: <unknown function> + 0x1688cb (0x5603f89438cb in /opt/conda/bin/python)
frame #16: _PyGC_CollectNoFail + 0x2a (0x5603f89cb79a in /opt/conda/bin/python)
frame #17: PyImport_Cleanup + 0x278 (0x5603f897ffa8 in /opt/conda/bin/python)
frame #18: Py_FinalizeEx + 0x61 (0x5603f89ea961 in /opt/conda/bin/python)
frame #19: Py_Main + 0x35e (0x5603f89f4cae in /opt/conda/bin/python)
frame #20: main + 0xee (0x5603f88bef2e in /opt/conda/bin/python)
frame #21: __libc_start_main + 0xe7 (0x7fc7f2cf3b97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #22: <unknown function> + 0x1c327f (0x5603f899e27f in /opt/conda/bin/python)

terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: device-side assert triggered
Exception raised from create_event_internal at ../c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fa371cb999b in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xc10 (0x7fa371efc280 in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7fa371ca1dfd in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #3: <unknown function> + 0x5414e2 (0x7fa3aad194e2 in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #4: <unknown function> + 0x19aaae (0x5559699ffaae in /opt/conda/bin/python)
frame #5: <unknown function> + 0xf2868 (0x555969957868 in /opt/conda/bin/python)
frame #6: <unknown function> + 0x1f0d91 (0x555969a55d91 in /opt/conda/bin/python)
frame #7: <unknown function> + 0xf270d (0x55596995770d in /opt/conda/bin/python)
frame #8: <unknown function> + 0x19aa90 (0x5559699ffa90 in /opt/conda/bin/python)
frame #9: <unknown function> + 0xf2868 (0x555969957868 in /opt/conda/bin/python)
frame #10: <unknown function> + 0x1f0d91 (0x555969a55d91 in /opt/conda/bin/python)
frame #11: <unknown function> + 0xf2828 (0x555969957828 in /opt/conda/bin/python)
frame #12: <unknown function> + 0x19aa90 (0x5559699ffa90 in /opt/conda/bin/python)
frame #13: <unknown function> + 0xf2868 (0x555969957868 in /opt/conda/bin/python)
frame #14: <unknown function> + 0x1f0d91 (0x555969a55d91 in /opt/conda/bin/python)
frame #15: <unknown function> + 0x1688cb (0x5559699cd8cb in /opt/conda/bin/python)
frame #16: _PyGC_CollectNoFail + 0x2a (0x555969a5579a in /opt/conda/bin/python)
frame #17: PyImport_Cleanup + 0x278 (0x555969a09fa8 in /opt/conda/bin/python)
frame #18: Py_FinalizeEx + 0x61 (0x555969a74961 in /opt/conda/bin/python)
frame #19: Py_Main + 0x35e (0x555969a7ecae in /opt/conda/bin/python)
frame #20: main + 0xee (0x555969948f2e in /opt/conda/bin/python)
frame #21: __libc_start_main + 0xe7 (0x7fa3d856fb97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #22: <unknown function> + 0x1c327f (0x555969a2827f in /opt/conda/bin/python)

terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: device-side assert triggered
Exception raised from create_event_internal at ../c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f121fb5299b in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xc10 (0x7f121fd95280 in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7f121fb3adfd in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #3: <unknown function> + 0x5414e2 (0x7f1258bb24e2 in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #4: <unknown function> + 0x19aaae (0x5601c5024aae in /opt/conda/bin/python)
frame #5: <unknown function> + 0xf2868 (0x5601c4f7c868 in /opt/conda/bin/python)
frame #6: <unknown function> + 0x1f0d91 (0x5601c507ad91 in /opt/conda/bin/python)
frame #7: <unknown function> + 0xf270d (0x5601c4f7c70d in /opt/conda/bin/python)
frame #8: <unknown function> + 0x19aa90 (0x5601c5024a90 in /opt/conda/bin/python)
frame #9: <unknown function> + 0xf2868 (0x5601c4f7c868 in /opt/conda/bin/python)
frame #10: <unknown function> + 0x1f0d91 (0x5601c507ad91 in /opt/conda/bin/python)
frame #11: <unknown function> + 0xf2828 (0x5601c4f7c828 in /opt/conda/bin/python)
frame #12: <unknown function> + 0x19aa90 (0x5601c5024a90 in /opt/conda/bin/python)
frame #13: <unknown function> + 0xf2868 (0x5601c4f7c868 in /opt/conda/bin/python)
frame #14: <unknown function> + 0x1f0d91 (0x5601c507ad91 in /opt/conda/bin/python)
frame #15: <unknown function> + 0x1688cb (0x5601c4ff28cb in /opt/conda/bin/python)
frame #16: _PyGC_CollectNoFail + 0x2a (0x5601c507a79a in /opt/conda/bin/python)
frame #17: PyImport_Cleanup + 0x278 (0x5601c502efa8 in /opt/conda/bin/python)
frame #18: Py_FinalizeEx + 0x61 (0x5601c5099961 in /opt/conda/bin/python)
frame #19: Py_Main + 0x35e (0x5601c50a3cae in /opt/conda/bin/python)
frame #20: main + 0xee (0x5601c4f6df2e in /opt/conda/bin/python)
frame #21: __libc_start_main + 0xe7 (0x7f1286408b97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #22: <unknown function> + 0x1c327f (0x5601c504d27f in /opt/conda/bin/python)

terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: device-side assert triggered
Exception raised from create_event_internal at ../c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fe94f54799b in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xc10 (0x7fe94f78a280 in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7fe94f52fdfd in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #3: <unknown function> + 0x5414e2 (0x7fe9885a74e2 in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #4: <unknown function> + 0x19aaae (0x55ab4542baae in /opt/conda/bin/python)
frame #5: <unknown function> + 0xf2868 (0x55ab45383868 in /opt/conda/bin/python)
frame #6: <unknown function> + 0x1f0d91 (0x55ab45481d91 in /opt/conda/bin/python)
frame #7: <unknown function> + 0xf270d (0x55ab4538370d in /opt/conda/bin/python)
frame #8: <unknown function> + 0x19aa90 (0x55ab4542ba90 in /opt/conda/bin/python)
frame #9: <unknown function> + 0xf2868 (0x55ab45383868 in /opt/conda/bin/python)
frame #10: <unknown function> + 0x1f0d91 (0x55ab45481d91 in /opt/conda/bin/python)
frame #11: <unknown function> + 0xf2828 (0x55ab45383828 in /opt/conda/bin/python)
frame #12: <unknown function> + 0x19aa90 (0x55ab4542ba90 in /opt/conda/bin/python)
frame #13: <unknown function> + 0xf2868 (0x55ab45383868 in /opt/conda/bin/python)
frame #14: <unknown function> + 0x1f0d91 (0x55ab45481d91 in /opt/conda/bin/python)
frame #15: <unknown function> + 0x1688cb (0x55ab453f98cb in /opt/conda/bin/python)
frame #16: _PyGC_CollectNoFail + 0x2a (0x55ab4548179a in /opt/conda/bin/python)
frame #17: PyImport_Cleanup + 0x278 (0x55ab45435fa8 in /opt/conda/bin/python)
frame #18: Py_FinalizeEx + 0x61 (0x55ab454a0961 in /opt/conda/bin/python)
frame #19: Py_Main + 0x35e (0x55ab454aacae in /opt/conda/bin/python)
frame #20: main + 0xee (0x55ab45374f2e in /opt/conda/bin/python)
frame #21: __libc_start_main + 0xe7 (0x7fe9b5dfdb97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #22: <unknown function> + 0x1c327f (0x55ab4545427f in /opt/conda/bin/python)

terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: device-side assert triggered
Exception raised from create_event_internal at ../c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7fce50e8399b in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xc10 (0x7fce510c6280 in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7fce50e6bdfd in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #3: <unknown function> + 0x5414e2 (0x7fce89ee34e2 in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #4: <unknown function> + 0x19aaae (0x55919a5ffaae in /opt/conda/bin/python)
frame #5: <unknown function> + 0xf2868 (0x55919a557868 in /opt/conda/bin/python)
frame #6: <unknown function> + 0x1f0d91 (0x55919a655d91 in /opt/conda/bin/python)
frame #7: <unknown function> + 0xf270d (0x55919a55770d in /opt/conda/bin/python)
frame #8: <unknown function> + 0x19aa90 (0x55919a5ffa90 in /opt/conda/bin/python)
frame #9: <unknown function> + 0xf2868 (0x55919a557868 in /opt/conda/bin/python)
frame #10: <unknown function> + 0x1f0d91 (0x55919a655d91 in /opt/conda/bin/python)
frame #11: <unknown function> + 0xf2828 (0x55919a557828 in /opt/conda/bin/python)
frame #12: <unknown function> + 0x19aa90 (0x55919a5ffa90 in /opt/conda/bin/python)
frame #13: <unknown function> + 0xf2868 (0x55919a557868 in /opt/conda/bin/python)
frame #14: <unknown function> + 0x1f0d91 (0x55919a655d91 in /opt/conda/bin/python)
frame #15: <unknown function> + 0x1688cb (0x55919a5cd8cb in /opt/conda/bin/python)
frame #16: _PyGC_CollectNoFail + 0x2a (0x55919a65579a in /opt/conda/bin/python)
frame #17: PyImport_Cleanup + 0x278 (0x55919a609fa8 in /opt/conda/bin/python)
frame #18: Py_FinalizeEx + 0x61 (0x55919a674961 in /opt/conda/bin/python)
frame #19: Py_Main + 0x35e (0x55919a67ecae in /opt/conda/bin/python)
frame #20: main + 0xee (0x55919a548f2e in /opt/conda/bin/python)
frame #21: __libc_start_main + 0xe7 (0x7fceb7739b97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #22: <unknown function> + 0x1c327f (0x55919a62827f in /opt/conda/bin/python)

terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: device-side assert triggered
Exception raised from create_event_internal at ../c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f01ad8c799b in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xc10 (0x7f01adb0a280 in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7f01ad8afdfd in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #3: <unknown function> + 0x5414e2 (0x7f01e69274e2 in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #4: <unknown function> + 0x19aaae (0x55c9bc565aae in /opt/conda/bin/python)
frame #5: <unknown function> + 0xf2868 (0x55c9bc4bd868 in /opt/conda/bin/python)
frame #6: <unknown function> + 0x1f0d91 (0x55c9bc5bbd91 in /opt/conda/bin/python)
frame #7: <unknown function> + 0xf270d (0x55c9bc4bd70d in /opt/conda/bin/python)
frame #8: <unknown function> + 0x19aa90 (0x55c9bc565a90 in /opt/conda/bin/python)
frame #9: <unknown function> + 0xf2868 (0x55c9bc4bd868 in /opt/conda/bin/python)
frame #10: <unknown function> + 0x1f0d91 (0x55c9bc5bbd91 in /opt/conda/bin/python)
frame #11: <unknown function> + 0xf2828 (0x55c9bc4bd828 in /opt/conda/bin/python)
frame #12: <unknown function> + 0x19aa90 (0x55c9bc565a90 in /opt/conda/bin/python)
frame #13: <unknown function> + 0xf2868 (0x55c9bc4bd868 in /opt/conda/bin/python)
frame #14: <unknown function> + 0x1f0d91 (0x55c9bc5bbd91 in /opt/conda/bin/python)
frame #15: <unknown function> + 0x1688cb (0x55c9bc5338cb in /opt/conda/bin/python)
frame #16: _PyGC_CollectNoFail + 0x2a (0x55c9bc5bb79a in /opt/conda/bin/python)
frame #17: PyImport_Cleanup + 0x278 (0x55c9bc56ffa8 in /opt/conda/bin/python)
frame #18: Py_FinalizeEx + 0x61 (0x55c9bc5da961 in /opt/conda/bin/python)
frame #19: Py_Main + 0x35e (0x55c9bc5e4cae in /opt/conda/bin/python)
frame #20: main + 0xee (0x55c9bc4aef2e in /opt/conda/bin/python)
frame #21: __libc_start_main + 0xe7 (0x7f021417db97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #22: <unknown function> + 0x1c327f (0x55c9bc58e27f in /opt/conda/bin/python)

terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: device-side assert triggered
Exception raised from create_event_internal at ../c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7ff569f1599b in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xc10 (0x7ff56a158280 in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7ff569efddfd in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #3: <unknown function> + 0x5414e2 (0x7ff5a2f754e2 in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #4: <unknown function> + 0x19aaae (0x562bbdb46aae in /opt/conda/bin/python)
frame #5: <unknown function> + 0xf2868 (0x562bbda9e868 in /opt/conda/bin/python)
frame #6: <unknown function> + 0x1f0d91 (0x562bbdb9cd91 in /opt/conda/bin/python)
frame #7: <unknown function> + 0xf270d (0x562bbda9e70d in /opt/conda/bin/python)
frame #8: <unknown function> + 0x19aa90 (0x562bbdb46a90 in /opt/conda/bin/python)
frame #9: <unknown function> + 0xf2868 (0x562bbda9e868 in /opt/conda/bin/python)
frame #10: <unknown function> + 0x1f0d91 (0x562bbdb9cd91 in /opt/conda/bin/python)
frame #11: <unknown function> + 0xf2828 (0x562bbda9e828 in /opt/conda/bin/python)
frame #12: <unknown function> + 0x19aa90 (0x562bbdb46a90 in /opt/conda/bin/python)
frame #13: <unknown function> + 0xf2868 (0x562bbda9e868 in /opt/conda/bin/python)
frame #14: <unknown function> + 0x1f0d91 (0x562bbdb9cd91 in /opt/conda/bin/python)
frame #15: <unknown function> + 0x1688cb (0x562bbdb148cb in /opt/conda/bin/python)
frame #16: _PyGC_CollectNoFail + 0x2a (0x562bbdb9c79a in /opt/conda/bin/python)
frame #17: PyImport_Cleanup + 0x278 (0x562bbdb50fa8 in /opt/conda/bin/python)
frame #18: Py_FinalizeEx + 0x61 (0x562bbdbbb961 in /opt/conda/bin/python)
frame #19: Py_Main + 0x35e (0x562bbdbc5cae in /opt/conda/bin/python)
frame #20: main + 0xee (0x562bbda8ff2e in /opt/conda/bin/python)
frame #21: __libc_start_main + 0xe7 (0x7ff5d07cbb97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #22: <unknown function> + 0x1c327f (0x562bbdb6f27f in /opt/conda/bin/python)

terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: device-side assert triggered
Exception raised from create_event_internal at ../c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) + 0x6b (0x7f9808d0299b in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xc10 (0x7f9808f45280 in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7f9808ceadfd in /opt/conda/lib/python3.6/site-packages/torch/lib/libc10.so)
frame #3: <unknown function> + 0x5414e2 (0x7f9841d624e2 in /opt/conda/lib/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #4: <unknown function> + 0x19aaae (0x55ba33d58aae in /opt/conda/bin/python)
frame #5: <unknown function> + 0xf2868 (0x55ba33cb0868 in /opt/conda/bin/python)
frame #6: <unknown function> + 0x1f0d91 (0x55ba33daed91 in /opt/conda/bin/python)
frame #7: <unknown function> + 0xf270d (0x55ba33cb070d in /opt/conda/bin/python)
frame #8: <unknown function> + 0x19aa90 (0x55ba33d58a90 in /opt/conda/bin/python)
frame #9: <unknown function> + 0xf2868 (0x55ba33cb0868 in /opt/conda/bin/python)
frame #10: <unknown function> + 0x1f0d91 (0x55ba33daed91 in /opt/conda/bin/python)
frame #11: <unknown function> + 0xf2828 (0x55ba33cb0828 in /opt/conda/bin/python)
frame #12: <unknown function> + 0x19aa90 (0x55ba33d58a90 in /opt/conda/bin/python)
frame #13: <unknown function> + 0xf2868 (0x55ba33cb0868 in /opt/conda/bin/python)
frame #14: <unknown function> + 0x1f0d91 (0x55ba33daed91 in /opt/conda/bin/python)
frame #15: <unknown function> + 0x1688cb (0x55ba33d268cb in /opt/conda/bin/python)
frame #16: _PyGC_CollectNoFail + 0x2a (0x55ba33dae79a in /opt/conda/bin/python)
frame #17: PyImport_Cleanup + 0x278 (0x55ba33d62fa8 in /opt/conda/bin/python)
frame #18: Py_FinalizeEx + 0x61 (0x55ba33dcd961 in /opt/conda/bin/python)
frame #19: Py_Main + 0x35e (0x55ba33dd7cae in /opt/conda/bin/python)
frame #20: main + 0xee (0x55ba33ca1f2e in /opt/conda/bin/python)
frame #21: __libc_start_main + 0xe7 (0x7f986f5b8b97 in /lib/x86_64-linux-gnu/libc.so.6)
frame #22: <unknown function> + 0x1c327f (0x55ba33d8127f in /opt/conda/bin/python)

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


Small inconsistency in tokenization_utils for special tokens retrieval

Hi there,

This is just a minor issue I have spotted.
When a special token (cls_token, sep_token, etc.) is accessed in an instantiated Tokenizer, this is the piece of code executed to retrieve the property:

```python
    @property
    def eos_token(self) -> str:
        """
        :obj:`str`: End of sentence token. Log an error if used while not having been set.
        """
        if self._eos_token is None and self.verbose:
            logger.error("Using eos_token, but it is not set yet.")
            return None
        return str(self._eos_token)
```

The None check is tied to the verbose flag, so when the verbose flag is set to False the condition is not triggered, and even if the special token is None, a 'None' literal string is returned (the last line).
It happens the same for all the special tokens, leading to unexpected behavior if you are expecting an actual None outside the tokenizer. I think that the "verbose" flag and the "is None" should be checked in separated conditionals.

The mentioned code can be located at:
https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_utils_base.py#L949

Thank you very much.


How to train encoder decoder for explicit negation generation

Hi
I am trying to generate negations out of non-negated sentences.
I used a simple “I have tea” => “I don’t have tea” formatted dataset for training an XLMR encoder-decoder model using the example provided in the collab.

```
# set special tokens

roberta_shared.config.decoder_start_token_id = tokenizer.bos_token_id

roberta_shared.config.eos_token_id = tokenizer.eos_token_id

# sensible parameters for beam search

# set decoding params

roberta_shared.config.max_length = 64

roberta_shared.config.early_stopping = True

roberta_shared.config.no_repeat_ngram_size = 3

roberta_shared.config.length_penalty = 2.0

roberta_shared.config.num_beams = 4

roberta_shared.config.vocab_size = roberta_shared.config.encoder.vocab_size

```

But the test set produces different tokens than the source. How can I preserve the source tokens when generating the output.
[“I have it.”, “I love tea”, “I can have coffee.”] =>
[‘I have no it.’, “I’ll not love.” “I can’t have food.”]
Where the model modifies the words in the sentence.


Fix: typo in FINE_TUNE_XLSR_WAV2VEC2.md

Fix typo.

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [x] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

This is a simple typo fix. Could you review it @sgugger ?

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


GPT Neo

# What does this PR do?

This PR adds the [GPT Neo model](https://github.com/EleutherAI/gpt-neo).
The model architecture is very similar to GPT2 except it local attention in alternate layers

- `LocalAttention` module implements the local attention. The implementation is not as clean as it should be and will be cleaned-up in follow-up PR.
- To enable caching (`use_cache`) the local attention layer caches the `hidden_states` instead of `past_key_value_states`.
Also right now when `use_cache` is enabled the current length can-not be greater than 1.
- The model uses the same tokenizer as GPT2 so does not need a new tokenizer class.

Example: usage
```python

import torch
from transformers import GPTNeoForCausalLM, AutoTokenizer

model = GPTNeoForCausalLM.from_pretrained("EleutherAI/gpt-neo-1.3B")
tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neo-1.3B")

unicorns = "In a shocking finding, scientists discovered a herd of unicorns living in a remote, " \
           "previously unexplored valley, in the Andes Mountains. Even more surprising to the " \
           "researchers was the fact that the unicorns spoke perfect English."

input_ids = tokenizer(unicorns, return_tensors="pt").input_ids

# add the length of the prompt tokens to match with the mesh-tf generation
max_length = 400 + input_ids.shape[1]

temperature = .9
do_sample = True

# set seed to reproduce samples
torch.manual_seed(42)

gen_tokens = model.generate(
  input_ids,
  do_sample=do_sample,
  min_length=max_length,
  max_length=max_length,
  temperature=temperature,
)
gen_text = tokenizer.batch_decode(gen_tokens)[0]
```

Future TODOs:
- clean-up the implementation of `LocalAttention` especially the creation of `attention_mask`.
- test fine-tuning.
- enable current length > 1 when `use_cache` is enabled.
- Add more robust and aggressive tests for the `LocalAttention` module.
- Add `TF` model.


fix code quality issues

### Description

Hi :wave: I sent PR #8950 which has changes in a lot of files so sending this PR with fixes in few files so that it can be easily reviewed.

You can have a look at the various issues that were caught in the codebase [here](https://deepsource.io/gh/withshubh/transformers/issues/?category=recommended).

### Summary of changes

- Removed length check in favour of truthiness of the object
> Boosts minor performance, see the description [here](https://deepsource.io/gh/withshubh/transformers/issue/PYL-C1801/description/).
- Removed unnecessary comprehension
> boosts minor performance, see the description [here](https://deepsource.io/gh/withshubh/transformers/issue/PTC-W0016/description/).
- Removed unnecessary use of comprehension
> boosts minor performance, see the description [here](https://deepsource.io/gh/withshubh/transformers/issue/PTC-W0019/description/).
- Refactored the comparison involving `not`
> fixed [antipattern](https://deepsource.io/gh/withshubh/transformers/issue/PYL-C0113/description/)
- Removed unnecessary return statement
> removes [antipattern](https://deepsource.io/gh/withshubh/transformers/issue/PYL-R1711/description/)
- Iterated dictionary directly
> removes [antipattern](https://deepsource.io/gh/withshubh/transformers/issue/PYL-C0201/description/)
- Used literal syntax instead of function calls to create data structure
> boosts minor [performance](https://deepsource.io/gh/withshubh/transformers/issue/PYL-C1801/description/)
- Added .deepsource.toml
> config file to continuously analyze the repo for code quality issues


[Wav2Vec2] Small tab fix

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Option to change loss function for fine tuning

# 🚀 Feature request

## Motivation

I was working in a multi class text classification problem for which I was using `DistilBertForSequenceClassification` and I found out that there is no way for me to change the loss function from CrossEntropyLoss.

## Your contribution

I can submit a PR, if this feature request is approved.


Add GPT-Neo

# 🌟 New model addition

Please add GPT-Neo

## Model description

> GPT-Neo is the code name for a series of transformer-based language models loosely styled around the GPT architecture that Eleuther AI plans to train and open source. Eleuther AI's primary goal is to replicate a GPT-3 sized model and open source it to the public, for free.

<!-- Important information -->

## Open source status

* [x] the model implementation is available: [Repo](https://github.com/EleutherAI/gpt-neo)
* [x] the model weights are available: [Download](https://the-eye.eu/eleuther_staging/gptneo-release/) (1.3B & 2.7B)
* [x] who are the authors: @sdtblck, @leogao2, @lucidrains, @ConnorJL, @StellaAthena & [others](https://github.com/EleutherAI)

Somewhat related to #4658, #4679, especially [this](https://github.com/huggingface/transformers/issues/4658#issuecomment-754247106)


Is there a `DataCollator` cat mask n-gram words for LM?

# 📚 Migration

## Information

I want a mask n-gram words when pre-train bert model? but I can't find  a DataCollator from the lib
https://github.com/huggingface/transformers/blob/master/src/transformers/data/data_collator.py

I want to build it by myself, but I don't know how to build my own DataCollator, who can give me a demo?

## Checklist

- [ ] I have read the migration guide in the readme.
 ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers);
  [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [ ] I checked if a related official extension example runs on my machine.


How to fine-tune RAG on MS-MARCO dataset?


issue of run_mlm.py

hello guys, I try to finetune bert in my own dataset (line by line txt, language:Chinese), follow the guid code of run_mlm.py example.
The tokenizer is bert pretrained tokenizer (tokenizer=AutoTokenizer.from_pretrained('bert-base-chinese')), and model is bert-base-chinese ,as folllows:
    config=BertConfig.from_pretrained('bert-base-chinese')
    print(config)
    model=BertForMaskedLM(config=config)
when I started the trainer, I got the following errors:

Using custom data configuration default-cd6deed448eea358
Downloading and preparing dataset text/default (download: Unknown size, generated: Unknown size, post-processed: Unknown size, total: Unknown size) to /home/hcl/.cache/huggingface/datasets/text/default-cd6deed448eea358/0.0.0/293ecb642f9fca45b44ad1f90c8445c54b9d80b95ab3fca3cfa5e1e3d85d4a57...
                                Dataset text downloaded and prepared to /home/hcl/.cache/huggingface/datasets/text/default-cd6deed448eea358/0.0.0/293ecb642f9fca45b44ad1f90c8445c54b9d80b95ab3fca3cfa5e1e3d85d4a57. Subsequent calls will reuse this data.
100%|██████████| 3264/3264 [01:06<00:00, 48.83ba/s]
  0%|          | 0/3264 [00:00<?, ?ba/s]
Traceback (most recent call last):
  File "/home/hcl/miniconda3/envs/pytorch/lib/python3.7/site-packages/datasets/arrow_dataset.py", line 1582, in _map_single
    writer.write_batch(batch)
  File "/home/hcl/miniconda3/envs/pytorch/lib/python3.7/site-packages/datasets/arrow_writer.py", line 276, in write_batch
    pa_table = pa.Table.from_pydict(typed_sequence_examples)
  File "pyarrow/table.pxi", line 1559, in pyarrow.lib.Table.from_pydict
  File "pyarrow/array.pxi", line 331, in pyarrow.lib.asarray
  File "pyarrow/array.pxi", line 222, in pyarrow.lib.array
  File "pyarrow/array.pxi", line 110, in pyarrow.lib._handle_arrow_array_protocol
  File "/home/hcl/miniconda3/envs/pytorch/lib/python3.7/site-packages/datasets/arrow_writer.py", line 98, in __arrow_array__
    if trying_type and out[0].as_py() != self.data[0]:
  File "pyarrow/array.pxi", line 1067, in pyarrow.lib.Array.__getitem__
  File "pyarrow/array.pxi", line 549, in pyarrow.lib._normalize_index
IndexError: index out of bounds
python-BaseException

for now, I don't know why, the first epoch runs well.
Any helps?
ps: my os is deepin15; I choose nvidia rtx2080ti as my gpu, and finetune my dataset on pytorch1.6


why My Albert pretrain loss can't decrease?

# 📚 Migration

## Information

<!-- Important information -->

Model i am using Albert

Language I am using the model on just digits(Desensitized Chinese)

The problem arises when using: Albert Trainer

## Details
The loss can decrease normal when I take this config to RoBerta, I use Albert replace RoBerta, The loss can't decrease, I don't know what's the problem, please help

```
%%time
bert_file = './albert'
from transformers import Trainer, TrainingArguments
from transformers import LineByLineTextDataset, DataCollatorForLanguageModeling
from transformers import AlbertConfig, AlbertForMaskedLM

config = AlbertConfig(
    hidden_size = 768,
    num_attention_heads = 12,
    intermediate_size = 3072,
    vocab_size = vocab_size + 10
)
model = AlbertForMaskedLM(config=config)

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)

train_dataset = LineByLineTextDataset(
    tokenizer=tokenizer,
    file_path="./my_wiki_file",
    block_size=128,
)

training_args = TrainingArguments(
    output_dir=bert_file,
    overwrite_output_dir=True,
    num_train_epochs=40,
    per_device_train_batch_size=64,
    save_steps=100000,
    save_total_limit=2,
    prediction_loss_only=False,
)

%%time
trainer = Trainer(
    model = model,
    args = training_args,
    data_collator = data_collator,
    train_dataset = train_dataset
)
trainer.train()
```
This is result
```
Step	Training Loss
500	6.687300
1000	4.034700
1500	3.826200
2000	3.777200
2500	3.788800
3000	3.751100
3500	3.780000
4000	3.772900
4500	3.795800
5000	3.737000
5500	3.782300
6000	3.775600
6500	3.821400
7000	3.730200
7500	3.751700
8000	3.787000
8500	3.824500
9000	3.746300
9500	3.782600
10000	3.770600
```

## Environment info
- `transformers` version:4.5.0.dev0
- Platform:Kaggle notebook
- Python version:3.7
- PyTorch version (GPU?):
- torch version: 1.8.0
- Using GPU in script?:YES
- Using distributed or parallel set-up in script?:NO

## Checklist

- [ ] I have read the migration guide in the readme.
 ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers);
  [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [ ] I checked if a related official extension example runs on my machine.


Fix on_step_begin and on_step_end Callback Sequencing

# What does this PR do?

Currently, the Trainer exhibits the following behavior (simplified):

```
for step, input in epoch_iterator:
     if (step + 1) % self.args.gradient_accumulation_steps == 0:
          callback_handler.on_step_begin()

     ...
     if (step + 1) % self.args.gradient_accumulation_steps == 0:
          # Apply Gradient Update (Finished accumulating)
          optimizer.step()
          callback_handler.on_step_end()
```

Unfortunately, this means that `on_step_begin()` gets called during the same iteration, *before* `on_step_end()` which is incorrect, and confuses folks implementing custom callbacks for timing individual iterations (like my team!).

Instead, this fix starts by calling `on_step_begin()` at steps = 0 (iteration 0) and will only be called on the next step after `on_step_end()`.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

Code updates part of the Trainer, so tagging @sgugger.


Can’t download the pre-trained pegasus-large model

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.2
- Platform:
- Python version: 3.7.10
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.4.1
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?:

## Who can help
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj

## Information

It appears that the huggingface.co model url has some problem.

Code:
```
from transformers import AutoTokenizer, TFAutoModelForSeq2SeqLM
tokenizer2 = AutoTokenizer.from_pretrained("google/pegasus-large")
model2 = TFAutoModelForSeq2SeqLM.from_pretrained("google/pegasus-large")
inputs1_2 = tokenizer2.encode("summarize: " + text1, return_tensors="tf", max_length=1024)
outputs1_2 = model2.generate(inputs1_2, max_length=150, min_length=40, length_penalty=2.0, num_beams=4, early_stopping=True)
outputs1_2, tokenizer2.decode(outputs1_2[0])

```
error message:
```
404 Client Error: Not Found for url: https://huggingface.co/google/pegasus-large/resolve/main/tf_model.h5

…

OSError: Can't load weights for 'google/pegasus-large'. Make sure that:

- 'google/pegasus-large' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'google/pegasus-large' is the correct path to a directory containing a file named one of tf_model.h5, pytorch_model.bin.

```


pegasus-xsum summarized a story of Eiffel Tower into one on the World Trade Center

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.2
- Platform:
- Python version: 3.7.10
- PyTorch version (GPU?):
- Tensorflow version (GPU?): 2.4.1
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?:

## Who can help
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj

## Information

The wired thing happened when I tried model pegasus-xsum for text summarization using the example code and data. I noticed that the output describes a similar but obviously different story than the one in the input. I expected to see some description of the Eiffel Tower, but the output is all about New York's World Trade Center!!

I noticed that the online demo version works fine, and the summary output is still on the Eiffel Tower.
https://huggingface.co/google/pegasus-xsum

It appears that pegasus-xsum model in my code generated the summary from some training data, but not the input I gave (retained memory?). How can I git the model behave normally like the online version?

The code I used (adopted from the online demo page):
```
text1="The tower is 324 metres (1,063 ft) tall, about the same height as an 81-storey building, and the tallest structure in Paris. Its base is square, measuring 125 metres (410 ft) on each side. During its construction, the Eiffel Tower surpassed the Washington Monument to become the tallest man-made structure in the world, a title it held for 41 years until the Chrysler Building in New York City was finished in 1930. It was the first structure to reach a height of 300 metres. Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft). Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct."

from transformers import AutoTokenizer, TFAutoModelForSeq2SeqLM
tokenizer1 = AutoTokenizer.from_pretrained("google/pegasus-xsum")
model1 = TFAutoModelForSeq2SeqLM.from_pretrained("google/pegasus-xsum")
inputs1_1 = tokenizer1.encode("summarize: " + text1, return_tensors="tf", max_length=1024)
outputs1_1 = model1.generate(inputs1_1, max_length=150, min_length=40, length_penalty=2.0, num_beams=4, early_stopping=True)
tokenizer1.decode(outputs1_1[0])
```

the output I got:
`"<pad> New York's World Trade Center is the tallest building in the United States and one of the world's tallest structures, with a total height of 1,776ft (541m), according to Guinness World Records."`


Generating text with MBart Large 50 on GPU with Tensorflow is significantly slower than with Pytorch

This applies to the [MBart Large MMT 50-language model](https://huggingface.co/facebook/mbart-large-50-many-to-many-mmt). Takes about 1m20s to process 10 batches of size 16 x 171 on pytorch, but 8min for tensorflow. Both are running on P100 through Kaggle.

- [Tensorflow notebook](https://www.kaggle.com/xhlulu/tf-mbartforconditionalgeneration-speed-test)
- [Pytorch Notebook](https://www.kaggle.com/xhlulu/mbartforconditionalgeneration-speed-test)

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.2
- Platform: Linux-5.4.89+-x86_64-with-debian-buster-sid
- Python version: 3.7.9
- PyTorch version (GPU?): 1.7.0 (True)
- Tensorflow version (GPU?): 2.4.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

## Code for pytorch

```python
from tqdm.auto import tqdm
import torch
from transformers import MBartForConditionalGeneration, MBart50TokenizerFast

article_hi = "संयुक्त राष्ट्र के प्रमुख का कहना है कि सीरिया में कोई सैन्य समाधान नहीं है"

model = MBartForConditionalGeneration.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")
tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")

# translate Hindi to French
tokenizer.src_lang = "hi_IN"

for i in tqdm(range(10)):
    encoded_hi = tokenizer([article_hi*10]*16, return_tensors="pt")

    generated_tokens = model.generate(
        **encoded_hi,
        forced_bos_token_id=tokenizer.lang_code_to_id["fr_XX"]
    )
    out = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)[0]
    # => "Le chef de l 'ONU affirme qu 'il n 'y a pas de solution militaire dans la Syrie."

```

## Code for Tensorflow

```python
from tqdm.auto import tqdm
import tensorflow as tf
from transformers import TFMBartForConditionalGeneration, MBart50TokenizerFast

strategy = tf.strategy.get_strategy()

article_hi = "संयुक्त राष्ट्र के प्रमुख का कहना है कि सीरिया में कोई सैन्य समाधान नहीं है"

with strategy.scope():
    model = TFMBartForConditionalGeneration.from_pretrained("facebook/mbart-large-50-many-to-many-mmt", from_pt=True)

tokenizer = MBart50TokenizerFast.from_pretrained("facebook/mbart-large-50-many-to-many-mmt")

# translate Hindi to French
tokenizer.src_lang = "hi_IN"

with strategy.scope():
    for i in tqdm(range(10)):
        encoded_hi = tokenizer([article_hi*10]*16, return_tensors="tf")

        generated_tokens = model.generate(
            **encoded_hi,
            forced_bos_token_id=tokenizer.lang_code_to_id["fr_XX"]
        )
        out = tokenizer.batch_decode(generated_tokens, skip_special_tokens=True)[0]
        # => "Le chef de l 'ONU affirme qu 'il n 'y a pas de solution militaire dans la Syrie."

```


Issues finetuning MBART 50 many to many

- `transformers` version: Latest
- Platform:
- Python version: 1.8.0
- Using GPU in script?: Yes A100
- Using distributed or parallel set-up in script?: No

I am trying to finetune MBART50-many-to-many

```
python ./transformers/examples/seq2seq/run_translation.py \
    --model_name_or_path facebook/mbart-large-50-many-to-many-mmt \
    --do_train \
    --do_eval \
    --source_lang ru_RU \
    --target_lang en_XX \
    --train_file ./corpus_v2/train.json \
    --validation_file ./corpus_v2/valid.json \
    --output_dir /local/nlpswordfish/tuhin/mbart50/tst-translation \
    --per_device_train_batch_size=32 \
    --per_device_eval_batch_size=8 \
    --overwrite_output_dir \
    --predict_with_generate \
    --max_train_samples 51373 \
    --max_val_samples 6424 \
    --gradient_accumulation_steps 1\
    --num_train_epochs 8 \
    --save_strategy epoch \
    --evaluation_strategy epoch

```

Even though I explicitly pass Src lang as ru_RU and Target as en_XX I get an error and see my log.
I tried printing Src and Tgt language

```

 Assigning ['ar_AR', 'cs_CZ', 'de_DE', 'en_XX', 'es_XX', 'et_EE', 'fi_FI', 'fr_XX', 'gu_IN', 'hi_IN', 'it_IT', 'ja_XX', 'kk_KZ', 'ko_KR', 'lt_LT', 'lv_LV', 'my_MM', 'ne_NP', 'nl_XX', 'ro_RO', 'ru_RU', 'si_LK', 'tr_TR', 'vi_VN', 'zh_CN', 'af_ZA', 'az_AZ', 'bn_IN', 'fa_IR', 'he_IL', 'hr_HR', 'id_ID', 'ka_GE', 'km_KH', 'mk_MK', 'ml_IN', 'mn_MN', 'mr_IN', 'pl_PL', 'ps_AF', 'pt_XX', 'sv_SE', 'sw_KE', 'ta_IN', 'te_IN', 'th_TH', 'tl_XX', 'uk_UA', 'ur_PK', 'xh_ZA', 'gl_ES', 'sl_SI'] to the additional_special_tokens key of the tokenizer
 Src lang is  en_XX
 ids [250004]
 ids [2]
 loading weights file https://huggingface.co/facebook/mbart-large-50-many-to-many-mmt/resolve/main/pytorch_model.bin from cache at /home/tuhin.chakr/.cache/huggingface/transformers/e33fcda1a71396b8475e16e2fe1458cfa62c6013f8cb3787d6aa4364ec5251c6.d802a5ca7720894045dd2c9dcee6069d27aa92fbbe33f52b44d479538dc3ccc3
 All model checkpoint weights were used when initializing MBartForConditionalGeneration.

 All the weights of MBartForConditionalGeneration were initialized from the model checkpoint at facebook/mbart-large-50-many-to-many-mmt.
 If your task is similar to the task the model of the checkpoint was trained on, you can already use MBartForConditionalGeneration for predictions without further training.
 Tgt lang is  None
 self.prefix_tokens is [None]
 ids [None]
 Traceback (most recent call last):
   File "./transformers/examples/seq2seq/run_translation.py", line 564, in <module
     main()
   File "./transformers/examples/seq2seq/run_translation.py", line 403, in main
     train_dataset = train_dataset.map(
   File "/home/tuhin.chakr/yes/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1289, in map
     update_data = does_function_return_dict(test_inputs, test_indices)
   File "/home/tuhin.chakr/yes/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1260, in does_function_return_dict
     function(*fn_args, indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
   File "./transformers/examples/seq2seq/run_translation.py", line 384, in preprocess_function
     with tokenizer.as_target_tokenizer():
   File "/home/tuhin.chakr/yes/lib/python3.8/contextlib.py", line 113, in __enter__
     return next(self.gen)
   File "/home/tuhin.chakr/yes/lib/python3.8/site-packages/transformers/models/mbart/tokenization_mbart50_fast.py", line 242, in as_target_tokenizer
     self.set_tgt_lang_special_tokens(self.tgt_lang)
   File "/home/tuhin.chakr/yes/lib/python3.8/site-packages/transformers/models/mbart/tokenization_mbart50_fast.py", line 269, in set_tgt_lang_special_tokens
     prefix_tokens_str = self.convert_ids_to_tokens(self.prefix_tokens)
   File "/home/tuhin.chakr/yes/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 287, in convert_ids_to_tokens
     index = int(index)
 TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType'

```

Also as far I understand in many to many for finetuning it requires some separate processing  based on the paper which is missing ?

![image](https://user-images.githubusercontent.com/3104771/111915770-500c5600-8a4e-11eb-8454-77c959bed2b4.png)

What should be the data format. Additionally will u guys release a many to one model as well ? although many to one is a subset of many to many

@patrickvonplaten, @patil-suraj


Local Attention for GPT2

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

Our model uses local attention in some layers (i.e each position can only see the last k=256 tokens in every other layer). We would like to be able to specify this in the config on the model hub.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

Right now we can't integrate the 1.3B and 2.7B EleutherAI GPT models because local attention is not supported in transformers.


weird large memory usage of mbert model

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.4.2
- Platform:
- Python version: 3.7
- PyTorch version (GPU?): 1.8
- Tensorflow version (GPU?):
- Using GPU in script?: -
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- nlp datasets: [different repo](https://github.com/huggingface/nlp)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->
albert, bert, xlm: @LysandreJik

## Information

I am using mbert model, this is 110 M params, I am testing the pretrianing codes with mt5-small and compare this with mbert, mbert weirdly use a lot of memory, I need to reduce 1/2 of batch_size on the same machine I train my mt5-small model which is 3x larger than mbert model. This is weird that mbert while being 1/3 of mt5-small require large memory.
* I am using run_mlm command
* I run the codes on V100  GPU

## To reproduce

Steps to reproduce the behavior:
python  run_mlm.py     --model_name_or_path bert-base-multilingual-uncased     --dataset_name wikitext     --dataset_config_name wikitext-2-raw-v1     --do_train     --do_eval     --output_dir output --per_device_train_batch_size 88  --fp16 --max_seq_length 128

## Expected behavior

larger batch size should be possible with mbert


run_mlm.py: CUDA error: device-side assert triggered, THCTensorIndex

## Environment info

- `transformers` version: 4.4.2
- Platform: Linux
- Python version: Python 3.4.9
- PyTorch version (GPU?): 1.6.0+cu101
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: Yes
- GPU details: 4 GPUs V100 16GB

## Information
I am using Bert and Roberta. I'm try to train from scratch on Wikipedia dataset using your examples run_mlm and your dataset wikipedia (20200501.en)
Before using distributed set up, I was stacked on the first optimization step. Without distributed setup I was stack on first optimization steps or received the reported error. With distributed setup I always receive the reported error.

The problem arises when using:
* [x] the official example scripts: https://github.com/huggingface/transformers/blob/master/examples/language-modeling/run_mlm.py
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: MLM train from scratch Bert and Roberta
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
```
export CUDA_LAUNCH_BLOCKING=1
export TOKENIZERS_PARALLELISM=true
export OMP_NUM_THREADS=32

source /data/medioli/env/bin/activate

python3 -m torch.distributed.launch \
--nproc_per_node 4 run_mlm.py \
--dataset_name wikipedia \
--tokenizer_name roberta-base \
--model_type roberta \
--dataset_config_name 20200501.en \
--do_train \
--do_eval \
--learning_rate 1e-5 \
--num_train_epochs 5 \
--save_steps 5000 \
--output_dir /data/medioli/models/mlm/wikipedia_roberta_5ep_1e5_lbl \
--line_by_line \
--use_fast_tokenizer \
--logging_dir /data/medioli/models/mlm/wikipedia_roberta_5ep_1e5_lbl/runs \
--cache_dir /data/medioli/datasets/wikipedia/ \
--overwrite_output_dir \
```

## Errors and Output

Many errors like this:
```
/pytorch/aten/src/THC/THCTensorIndex.cu:272: indexSelectLargeIndex: block: [372,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed.
```
Then:
```
Traceback (most recent call last):
  File "/data/medioli/transformers/examples/language-modeling/run_mlm.py", line 491, in <module>
    main()
  File "/data/medioli/transformers/examples/language-modeling/run_mlm.py", line 457, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/data/medioli/env/lib/python3.6/site-packages/transformers/trainer.py", line 1053, in train
    tr_loss += self.training_step(model, inputs)
  File "/data/medioli/env/lib/python3.6/site-packages/transformers/trainer.py", line 1443, in training_step
    loss = self.compute_loss(model, inputs)
  File "/data/medioli/env/lib/python3.6/site-packages/transformers/trainer.py", line 1475, in compute_loss
    outputs = model(**inputs)
  File "/data/medioli/env/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/medioli/env/lib64/python3.6/site-packages/torch/nn/parallel/distributed.py", line 511, in forward
    output = self.module(*inputs[0], **kwargs[0])
  File "/data/medioli/env/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/medioli/env/lib/python3.6/site-packages/transformers/models/roberta/modeling_roberta.py", line 1057, in forward
    return_dict=return_dict,
  File "/data/medioli/env/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/medioli/env/lib/python3.6/site-packages/transformers/models/roberta/modeling_roberta.py", line 810, in forward
    past_key_values_length=past_key_values_length,
  File "/data/medioli/env/lib64/python3.6/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/data/medioli/env/lib/python3.6/site-packages/transformers/models/roberta/modeling_roberta.py", line 123, in forward
    embeddings += position_embeddings
RuntimeError: CUDA error: device-side assert triggered
terminate called after throwing an instance of 'c10::Error'
  what():  CUDA error: device-side assert triggered
Exception raised from create_event_internal at /pytorch/c10/cuda/CUDACachingAllocator.cpp:687 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7fa4517ed1e2 in /data/medioli/env/lib64/python3.6/site-packages/torch/lib/libc10.so)
frame #1: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0xad2 (0x7fa451a3bf92 in /data/medioli/env/lib64/python3.6/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::TensorImpl::release_resources() + 0x4d (0x7fa4517db9cd in /data/medioli/env/lib64/python3.6/site-packages/torch/lib/libc10.so)
frame #3: std::vector<c10d::Reducer::Bucket, std::allocator<c10d::Reducer::Bucket> >::~vector() + 0x25a (0x7fa427f8489a in /data/medioli/env/lib64/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #4: c10d::Reducer::~Reducer() + 0x28a (0x7fa427f79b1a in /data/medioli/env/lib64/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #5: std::_Sp_counted_ptr<c10d::Reducer*, (__gnu_cxx::_Lock_policy)2>::_M_dispose() + 0x12 (0x7fa427f593c2 in /data/medioli/env/lib64/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #6: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() + 0x46 (0x7fa4277577a6 in /data/medioli/env/lib64/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #7: <unknown function> + 0xa6b08b (0x7fa427f5a08b in /data/medioli/env/lib64/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #8: <unknown function> + 0x273c00 (0x7fa427762c00 in /data/medioli/env/lib64/python3.6/site-packages/torch/lib/libtorch_python.so)
frame #9: <unknown function> + 0x274e4e (0x7fa427763e4e in /data/medioli/env/lib64/python3.6/site-packages/torch/lib/libtorch_python.so)
<omitting python frames>
frame #22: main + 0x16e (0x400a3e in /data/medioli/env/bin/python3)
frame #23: __libc_start_main + 0xf5 (0x7fa48f4903d5 in /lib64/libc.so.6)
frame #24: /data/medioli/env/bin/python3() [0x400b02]
```
Discussion in pytorch: https://discuss.pytorch.org/t/solved-assertion-srcindex-srcselectdimsize-failed-on-gpu-for-torch-cat/1804/22

Who can help me
Models:
@LysandreJik

Library:
- tokenizers: @LysandreJik
- trainer: @sgugger

Examples:
- maintained examples (not research project or legacy): @sgugger, @patil-suraj


Encoder Decoder Model didn't return a reasonable result

Hello,
I tried the example code in the official website as below.

# code
`from transformers import EncoderDecoderModel, BertTokenizer
import torch
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased')
input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)
outputs = model(input_ids=input_ids, decoder_input_ids=input_ids)
outputs = model(input_ids=input_ids, decoder_input_ids=input_ids, labels=input_ids)
loss, logits = outputs.loss, outputs.logits
model.save_pretrained("bert2bert")
model = EncoderDecoderModel.from_pretrained("bert2bert")
generated = model.generate(input_ids, decoder_start_token_id=model.config.decoder.pad_token_id)

for i, sample_output in enumerate(generated):
  print("{}: {}".format(i, tokenizer.decode(sample_output, skip_special_tokens=True)))`
# output
However, it returned to such a result.

`Some weights of the model checkpoint at bert-base-uncased were not used when initializing BertLMHeadModel: ['cls.seq_relationship.weight', 'cls.seq_relationship.bias']
- This IS expected if you are initializing BertLMHeadModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertLMHeadModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertLMHeadModel were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['bert.encoder.layer.0.crossattention.self.query.weight', 'bert.encoder.layer.0.crossattention.self.query.bias', 'bert.encoder.layer.0.crossattention.self.key.weight', 'bert.encoder.layer.0.crossattention.self.key.bias', 'bert.encoder.layer.0.crossattention.self.value.weight', 'bert.encoder.layer.0.crossattention.self.value.bias', 'bert.encoder.layer.0.crossattention.output.dense.weight', 'bert.encoder.layer.0.crossattention.output.dense.bias', 'bert.encoder.layer.0.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.0.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.1.crossattention.self.query.weight', 'bert.encoder.layer.1.crossattention.self.query.bias', 'bert.encoder.layer.1.crossattention.self.key.weight', 'bert.encoder.layer.1.crossattention.self.key.bias', 'bert.encoder.layer.1.crossattention.self.value.weight', 'bert.encoder.layer.1.crossattention.self.value.bias', 'bert.encoder.layer.1.crossattention.output.dense.weight', 'bert.encoder.layer.1.crossattention.output.dense.bias', 'bert.encoder.layer.1.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.1.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.2.crossattention.self.query.weight', 'bert.encoder.layer.2.crossattention.self.query.bias', 'bert.encoder.layer.2.crossattention.self.key.weight', 'bert.encoder.layer.2.crossattention.self.key.bias', 'bert.encoder.layer.2.crossattention.self.value.weight', 'bert.encoder.layer.2.crossattention.self.value.bias', 'bert.encoder.layer.2.crossattention.output.dense.weight', 'bert.encoder.layer.2.crossattention.output.dense.bias', 'bert.encoder.layer.2.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.2.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.3.crossattention.self.query.weight', 'bert.encoder.layer.3.crossattention.self.query.bias', 'bert.encoder.layer.3.crossattention.self.key.weight', 'bert.encoder.layer.3.crossattention.self.key.bias', 'bert.encoder.layer.3.crossattention.self.value.weight', 'bert.encoder.layer.3.crossattention.self.value.bias', 'bert.encoder.layer.3.crossattention.output.dense.weight', 'bert.encoder.layer.3.crossattention.output.dense.bias', 'bert.encoder.layer.3.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.3.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.4.crossattention.self.query.weight', 'bert.encoder.layer.4.crossattention.self.query.bias', 'bert.encoder.layer.4.crossattention.self.key.weight', 'bert.encoder.layer.4.crossattention.self.key.bias', 'bert.encoder.layer.4.crossattention.self.value.weight', 'bert.encoder.layer.4.crossattention.self.value.bias', 'bert.encoder.layer.4.crossattention.output.dense.weight', 'bert.encoder.layer.4.crossattention.output.dense.bias', 'bert.encoder.layer.4.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.4.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.5.crossattention.self.query.weight', 'bert.encoder.layer.5.crossattention.self.query.bias', 'bert.encoder.layer.5.crossattention.self.key.weight', 'bert.encoder.layer.5.crossattention.self.key.bias', 'bert.encoder.layer.5.crossattention.self.value.weight', 'bert.encoder.layer.5.crossattention.self.value.bias', 'bert.encoder.layer.5.crossattention.output.dense.weight', 'bert.encoder.layer.5.crossattention.output.dense.bias', 'bert.encoder.layer.5.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.5.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.6.crossattention.self.query.weight', 'bert.encoder.layer.6.crossattention.self.query.bias', 'bert.encoder.layer.6.crossattention.self.key.weight', 'bert.encoder.layer.6.crossattention.self.key.bias', 'bert.encoder.layer.6.crossattention.self.value.weight', 'bert.encoder.layer.6.crossattention.self.value.bias', 'bert.encoder.layer.6.crossattention.output.dense.weight', 'bert.encoder.layer.6.crossattention.output.dense.bias', 'bert.encoder.layer.6.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.6.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.7.crossattention.self.query.weight', 'bert.encoder.layer.7.crossattention.self.query.bias', 'bert.encoder.layer.7.crossattention.self.key.weight', 'bert.encoder.layer.7.crossattention.self.key.bias', 'bert.encoder.layer.7.crossattention.self.value.weight', 'bert.encoder.layer.7.crossattention.self.value.bias', 'bert.encoder.layer.7.crossattention.output.dense.weight', 'bert.encoder.layer.7.crossattention.output.dense.bias', 'bert.encoder.layer.7.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.7.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.8.crossattention.self.query.weight', 'bert.encoder.layer.8.crossattention.self.query.bias', 'bert.encoder.layer.8.crossattention.self.key.weight', 'bert.encoder.layer.8.crossattention.self.key.bias', 'bert.encoder.layer.8.crossattention.self.value.weight', 'bert.encoder.layer.8.crossattention.self.value.bias', 'bert.encoder.layer.8.crossattention.output.dense.weight', 'bert.encoder.layer.8.crossattention.output.dense.bias', 'bert.encoder.layer.8.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.8.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.9.crossattention.self.query.weight', 'bert.encoder.layer.9.crossattention.self.query.bias', 'bert.encoder.layer.9.crossattention.self.key.weight', 'bert.encoder.layer.9.crossattention.self.key.bias', 'bert.encoder.layer.9.crossattention.self.value.weight', 'bert.encoder.layer.9.crossattention.self.value.bias', 'bert.encoder.layer.9.crossattention.output.dense.weight', 'bert.encoder.layer.9.crossattention.output.dense.bias', 'bert.encoder.layer.9.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.9.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.10.crossattention.self.query.weight', 'bert.encoder.layer.10.crossattention.self.query.bias', 'bert.encoder.layer.10.crossattention.self.key.weight', 'bert.encoder.layer.10.crossattention.self.key.bias', 'bert.encoder.layer.10.crossattention.self.value.weight', 'bert.encoder.layer.10.crossattention.self.value.bias', 'bert.encoder.layer.10.crossattention.output.dense.weight', 'bert.encoder.layer.10.crossattention.output.dense.bias', 'bert.encoder.layer.10.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.10.crossattention.output.LayerNorm.bias', 'bert.encoder.layer.11.crossattention.self.query.weight', 'bert.encoder.layer.11.crossattention.self.query.bias', 'bert.encoder.layer.11.crossattention.self.key.weight', 'bert.encoder.layer.11.crossattention.self.key.bias', 'bert.encoder.layer.11.crossattention.self.value.weight', 'bert.encoder.layer.11.crossattention.self.value.bias', 'bert.encoder.layer.11.crossattention.output.dense.weight', 'bert.encoder.layer.11.crossattention.output.dense.bias', 'bert.encoder.layer.11.crossattention.output.LayerNorm.weight', 'bert.encoder.layer.11.crossattention.output.LayerNorm.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
2021-03-21 16:47:27.243389: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'cudart64_101.dll'; dlerror: cudart64_101.dll not found
2021-03-21 16:47:27.243603: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
0: . as as as as as as as as as as as as as as as as as as

Process finished with exit code 0
`
# issue
Would you be too kindly to help me find out the reason why it returned word 'as' ? Much thanks!
Besides, as a newbie, would it be possible if I could use BERT as encoder and Transformer as Decoder in this EncoderDecoderModel? I would be too grateful if you could help me out!


getting nans with t5-large + fix

## Environment info

- `transformers` version: 4.5.0.dev0
- Platform: Linux-4.15.0-65-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyTorch version (GPU?): 1.7.1+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@patil-suraj @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): t5-large

The problem arises when using:
* [ ] my own modified scripts: run_seq2seq with minor modifications (attached)

The tasks I am working on is:
* [ ] my own task or dataset: Closed-Book Open Domain QA

## To reproduce

Steps to reproduce the behavior (the fix I'm suggesting is very simple, so perhaps there is no reason to reproduce):
1. unzip the attached zip (below).
2. run
```bash
python run_seq2seq.py --model_name_or_path=t5-large
--do_train
--do_eval
--task=qa
--train_file=data/PAQ.filtered.regular.16000.json
--validation_file=data/PAQ.filtered.regular.16000.json
--output_dir=results/5e-5-t5-large-4096000-128-140-1792000-0.1-regular-true-4
--overwrite_output_dir
--per_device_train_batch_size=1
--per_device_eval_batch_size=128
--predict_with_generate
--fp16
--max_steps=1000
--evaluation_strategy=steps
--text_column=question
--summary_column=answer
--save_total_limit=5
--cache_dir=../.cache
--save_steps=500000
--learning_rate=5e-5
--eval_steps=96000
--warmup_steps=100
--run_name=5e-5-t5-large-4096000-128-140-1792000-0.1-regular-true-4
--dropout_rate=0.1
--gradient_accumulation_steps=1
--logging_steps=1
```

## Expected behavior
Training without nans.

## Possible fix
I debugged and saw that we get nans at the `modeling_t5.py` script in line 241
```python
hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)
```
By modifing this line to:
```python
clamp_value = torch.finfo(hidden_states.dtype).max - 1000
hidden_states = torch.clamp(hidden_states, min=-clamp_value, max=clamp_value) * torch.rsqrt(variance + self.variance_epsilon)
```
It seems to be solved.

BTW it happens in the last layers (this might explain why it wasn't caught in [this fix](https://discuss.huggingface.co/t/t5-fp16-issue-is-fixed/3139))

[seq2seq.zip](https://github.com/huggingface/transformers/files/6177063/seq2seq.zip)


[Wav2Vec2] Small improvements for wav2vec2 info script


[wav2vec sprint doc] add doc for Local machine

# What does this PR do?

Add instructions for how to do Wav2Vec2 fine-tuning on local machine.


Log continuously models with wandb

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

wandb integration currently logs last model (which can be the best by using `TrainingArguments.load_best_model_at_end`).

It would be great to allow continuous upload of model with appropriate aliases to versions.

Options would be:
    * `WANDB_LOG_MODEL = True` which just logs at the end as currently (not sure if we want to add scheduler and optimizer)
    * `WANDB_LOG_MODEL = 'all'` which logs continuously the model
    * `WANDB_LOG_MODEL = False` which does not log the model

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

Training can be very long and it would be so sad to lose a model :sob:

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

I can probably propose a PR but would love brainstorming on the ideal logic:
1. should we leverage `Trainer.save_model` (as currently) or `Trainer._save_checkpoint`
2. should we consider an artifact version as containing only the model & config or also containing optimizer and scheduler? Or should it actually be 2 separate artifacts?
3. if we leverage `on_save`, can we avoid the same current logic (fake trainer saving to a temporary directory that is then uploaded async) and just use an actual copy of what has been saved. We would just need the path or list of files that have been saved (should be straightforward)
4. If we log continuously the model, should we upload it only if it's improved (when `metric_for_best_model` is defined)? If that's the case, we'll need to be able to detect when that is the case. If that's not the case we'll still need to be able to know which one is the best.


feat(wandb): logging and configuration improvements

# What does this PR do?

Following improvements to `wandb` integration:
* ensure unique artifact id → previously it was based on run name which could create duplicates and mismatches (as runs can be updated manually in the UI)
* allow manual calls to `wandb.init()` → previously it would have closed the run and started a new one
* when a wandb run already exists (manually created), adds automatically model config parameters
* simplify reinit logic → now explicitly closes a run for hp search and avoid use of reinit which can have complex side effects (different behavior in notebooks vs scripts)
* ensure we have no dropped values (when step is below a previous logged value) by logging the step as an independent `train/global_step` metric and set it as default x-axis in the UI (can be edited manually). Note: this auto-setting of x-axis will be activated in upcoming release of wandb
* get values committed immediately so they appear in the UI with no delay
* fixes compatibility with sagemaker

Fixes https://github.com/wandb/client/issues/1499, #8754, #10486

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [X] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

Documentation: @sgugger


ReformerEmbedding unclear behavior

## Environment info
- `transformers` version: 4.3.3
- Platform: Linux-4.15.0-136-generic-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
@patrickvonplaten
## Information

Model I am using (Bert, XLNet ...): Reformer

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: MNIST

## To reproduce

Steps to reproduce the behavior:

1. Leave max_position_embeddings undefined or different than axial_pos_embedding shape
2. Observe the assert get triggered for large sequence lengths (within limit of axial_pos_embedding) https://github.com/huggingface/transformers/blob/master/src/transformers/models/reformer/modeling_reformer.py#L255

## Expected behavior

* No assert

## Additional Details:

`max_position_embeddings` is only used by `PositionEmbeddings`, thus if we provide `axial_pos_embds`  in the configuration, the `max_position_embeddings` will be not used for positional embedding. Moreover it is considered for factorizing `num_buckets` in `LSHSelfAttention` layer.

Thus in a scenario of using axial positional embedding, both the assert check will be useless as well as the [bucket factorization](https://github.com/huggingface/transformers/blob/master/src/transformers/models/reformer/modeling_reformer.py#L711)


Running "convert_graph_to_onnx.py" doesn't work.

## Environment info
- `transformers` version: 4.5.0.dev0
- Platform: Darwin-19.6.0-x86_64-i386-64bit
- Python version: 3.7.2
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten, @patil-suraj

Models:
Bart (https://huggingface.co/transformers/model_doc/marian.html)

 -->

## Information

A problem arises when I run "python3 convert_graph_to_onnx.py" where I receive the following error message
```
Traceback (most recent call last):
  File "convert_graph_to_onnx.py", line 22, in <module>
    from .file_utils import ModelOutput, is_tf_available, is_torch_available
ModuleNotFoundError: No module named '__main__.file_utils'; '__main__' is not a package
```

## To reproduce
Run "python3 convert_graph_to_onnx.py" inside the following directory transformers/src/transformers.

Steps to reproduce the behavior:

1. cd "transformers/src/transformers/"
2. python3 convert_graph_to_onnx.py

## Expected behavior

I expect convert_graph_to_onnx.py to begin running.


Modify the Trainer class to handle simultaneous execution of Ray Tune and Weights & Biases

# What does this PR do?

The proper way to integrate Ray Tune and Weights & Biases is to pass a `wandb` parameter to `tune.run`.

However, this parameter is handled as a dictionary inside the `config` argument, and there is no distinction between `wandb` parameters and standard model optimization parameters. The following code comes from [their docs](https://docs.wandb.ai/integrations/ray-tune):

```python
from ray.tune.logger import DEFAULT_LOGGERS
from ray.tune.integration.wandb import WandbLogger
tune.run(
    train_fn,
    config={
        # define search space here
        "parameter_1": tune.choice([1, 2, 3]),
        "parameter_2": tune.choice([4, 5, 6]),
        # wandb configuration
        "wandb": {
            "project": "Optimization_Project",
            "api_key_file": "/path/to/file",
            "log_config": True
        }
    },
    loggers=DEFAULT_LOGGERS + (WandbLogger, ))
```

This is not a problem for Ray Tune. However, it is a problem for the `transformers` integration because it treats wandb as a model parameter, and therefore configuring wandb in this way will raise an error message claiming that `wandb is not a training argument`.

The following code will raise such an error:

```python
    # Initialize our Trainer
    trainer = Trainer(
        model_init=model_init,
        args=training_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset if training_args.do_eval else None,
        compute_metrics=compute_metrics,
        tokenizer=tokenizer,
        data_collator=data_collator,
    )

    # Hyperparameter Search

    def hp_space_fn(empty_arg):
        config = {
                    "warmup_steps": tune.choice([50, 100, 500, 1000]),
                    "learning_rate": tune.choice([1.5e-5, 2e-5, 3e-5, 4e-5]),
                    "num_train_epochs": tune.quniform(0.0, 10.0, 0.5),
        }
        wandb_config = {
                "wandb": {
                        "project": os.environ.get(
                            'WANDB_PROJECT',
                            'wandb_project'),
                        "api_key": os.environ.get('API_KEY'),
                        "log_config": True
                        }
        }
        config.update(wandb_config)
        return config

    best_run = trainer.hyperparameter_search(
            direction="maximize",
            backend="ray",
            scheduler=PopulationBasedTraining(
                        time_attr='time_total_s',
                        metric='eval_f1_thr_0',
                        mode='max',
                        perturbation_interval=600.0
                    ),
            hp_space=hp_space_fn,
            loggers=DEFAULT_LOGGERS + (WandbLogger, ),
    )
```

One way to work around this is to instantiate a subclass based on the Trainer:

```python
    class CustomTrainer(Trainer):

        def __init__(self, *args, **kwargs):
            super(CustomTrainer, self).__init__(*args, **kwargs)

        def _hp_search_setup(self, trial: Any):
            try:
                trial.pop('wandb', None)
            except AttributeError:
                pass
            super(CustomTrainer, self)._hp_search_setup(trial)
```

However, this looks like a hack because throwing away `wandb` arguments in model config on `_hp_search_setup` should be standard Trainer behavior.

That's why I'm submitting a PR that directly modifies the `_hp_search_setup` of the Trainer class to ignore `wandb` arguments if Ray is chosen as a backend.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

I'm tagging @richardliaw and @amogkam as they're directly involved in Ray Tune.