Importing unittests using python unittest framework

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2
- Platform: Linux-5.3.0-1034-azure-x86_64-with-debian-buster-sid
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.0a0+8f84ded (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->
@sgugger

## Information

I am trying to run unit tests after cloning fresh repo.  I use python unittest framework and try to load the tests using "unittest.defaultTestLoader". Most of the tests pass, but I get few errors. All the errors are same - "**cannot initialize type "_CudaDeviceProperties": an object with that name is already defined**". This problem goes away if I user python unittest command line .

The problem arises when using:
* [ ] my own modified scripts: (give details below)
I use following script to load tests.

```
import os
import glob
import unittest
test_files_path='./tests/*'
test_files = [os.path.basename(x) for x in glob.glob(test_files_path)]
module_strings = ['tests.'+test_file[0:len(test_file)-3] for test_file in test_files]
print(module_strings)
suites = [unittest.defaultTestLoader.loadTestsFromName(test_file) for test_file in module_strings]
test_suite = unittest.TestSuite(suites)
test_runner = unittest.TextTestRunner().run(test_suite)
```

## To reproduce

Steps to reproduce the behavior:
1. Paste above code in a python file under transformers directory.
2.Run python file.

## Expected behavior

All tests should pass since I just cloned a fresh repo.


ignore FutureWarning in tests

As discussed in https://github.com/huggingface/transformers/pull/7033 we can't deal with transformers' `FutureWarning` in tests, since we have to keep those tests around until they become normal warnings and then the tests will get fixed/adjusted. So currently they just generate noise that can't be acted upon.

The only side-effect I can see is with other libraries' FutureWarnings, which now will be silenced too, but again we can easily fix those as soon as those aren't futuristic any more.


[T5Tokenizer] remove prefix_tokens

Fixes #7077

@sshleifer


T5Tokenizer shouldn't add pad token as prefix to labels

## Information
`prepare_seq2seq_batch` method in `T5Tokenizer` now prefixes `pad` token to the `labels`. [here](https://github.com/huggingface/transformers/blob/master/src/transformers/tokenization_t5.py#L362)
But in finetune.py [here](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/finetune.py#L149) we are calling `_shift_right` for T5 , which again adds another `pad` token at the beginning, so now `decoder_input_ids` contain two `pad` tokens.
## To reproduce

```python
from transformers import T5Tokenizer, T5Model
model = T5Model.from_pretrained("t5-small")
tok = T5Tokenizer.from_pretrained("t5-small")
enc = tok.prepare_seq2seq_batch("src text", "target text", return_tensors="pt")
print(enc["labels"])
# tensor([[   0, 2387, 1499,    1]])
decoder_input_ids = model._shift_right(enc["labels"]) # call _shift_right
print(decoder_input_ids)
#tensor([[   0,    0, 2387, 1499]])
```
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
There should be no special prefix token for T5 `labels`

@sshleifer


some sshleifer/xsum hub models  have bart-large-cnn task_specific_params['summarization']

S3 only fix!


[Benchmarks] Change all args to from `no_...` to their positive form

Fixes #7072

As @stas00 pointed out a while back, there were a bunch of negative args formulations in `benchmarks.py` which are hard to understand and make the code unnecessarly complex.

This PR fixes this problem by
a) Formulating all args in their positive form
b) Improving their docstring
c) Making the code more readable

**!!! This has breaking changes for the command line args of `examples/run_benchmark.py `and `examples/run_benchmark_tf.py` !!!**
People that were using `--no_...` on command line previously, now should use `--no-...`. *E.g.* `--no_inference` is changed to `--no-inference` on the command line.

Thanks a lot @fmcurti for leading this PR!


Compute loss method

Make it easier to write a custom loss by isolating the loss computation in a separate method.

Also document how to write a Trainer using such a custom loss.


Add tests and fix various bugs in ModelOutput

Fix a few bugs in `ModelOutput` and adds tests, mainly:
- if there is only one attribute not `None`, it wasn't accessible by key (and recognized inside the dictionary)
- if an attribute was set with `model_output.att_name` = ..., the corresponding key in the dictionary was not updated
- if an attribute was set with `model_output["key"] = ...`, the corresponding attribute was not updated.


Clean up `benchmark_args_utils.py` "no_..." arguments

# 🚀 Feature request

Currently we have a mixture of negative and positive formulated arguments, *e.g.* `no_cuda` and `training` here: https://github.com/huggingface/transformers/blob/0054a48cdd64e7309184a64b399ab2c58d75d4e5/src/transformers/benchmark/benchmark_args_utils.py#L61.

We should change all arguments to be positively formulated, *e.g. from `no_cuda` to `cuda`. These arguments should then change their default value from `False` to `True`.

Also the help text should be updated to something that is better formulated: "Don't ...." as a help text is not very easy to understand.

The motivation is clear: It's better to be consistent in a library and have the code as easy and intuitive to understand.

## Your contribution

This is a "good first issue", so I'm happy to help anybody who wants to take a shot at this :-)


added bangla-bert-base model card and also modified other model cards

model card

Related to pretrained model cards

Hi,
I added model card for bangla-bert-base model and also modified some typo in other model cards also.
Please check and if possible please merge.

thanks and regards
Sagor


MobileBERT inconsistent output (padded / not padded text)

## Environment info
- `transformers` version: 3.0.2
- Platform: Linux-5.4.0-47-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.3
- PyTorch version (GPU?): 1.5.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@LysandreJik, @mfuntowicz

## Information

When using MobileBERT model `google/mobilebert-uncased`, I get different model outputs for the same text when it contains padding and when it does not contain any padding (is the longest text in the batch). The same code produces correct results for other models, such as `bert-base-uncased`, so the issue is probably only with MobileBERT. The problem happens for both `CLS` output and all features.

Model I am using (Bert, XLNet ...): MobileBERT

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce
```python
from transformers import AutoModel, AutoTokenizer

model = 'google/mobilebert-uncased'
tokenizer = AutoTokenizer.from_pretrained(model)
model = AutoModel.from_pretrained(model)
text = 'Hey, how are you?'

i1 = tokenizer.batch_encode_plus([text], padding=True, return_tensors='pt')
emb1 = model(**i1)[1][0]  # Only one in batch (not padded): [-2.5088e+07,  7.3279e+04,  1.7544e+05, ...]

i2 = tokenizer.batch_encode_plus([text, text + ' hey'], padding=True, return_tensors='pt')
emb2 = model(**i2)[1][0]  # Not longest (padded): [-2.4871e+07,  8.1873e+04,  1.6693e+05, ...]

i3 = tokenizer.batch_encode_plus([text, text[:-10]], padding=True, return_tensors='pt')
emb3 = model(**i3)[1][0]  # Longest in batch (not padded): [-2.5088e+07,  7.3281e+04,  1.7544e+05, ...]

i4 = tokenizer.encode(text, return_tensors='pt')
emb4 = model(i4)[1][0]  # Without attention masks (not padded): [-2.5088e+07,  7.3279e+04,  1.7544e+05, ...]

i5 = tokenizer.encode_plus(text, return_tensors='pt')
emb5 = model(**i5)[1][0]  # With attention masks (not padded): [-2.5088e+07,  7.3279e+04,  1.7544e+05, ...]
```

## Expected behavior

I would expect the model to always return the same embeddings, but I don't really know which of the two variants is correct.


How to use hugging on onw embedding

i already get a embedding，or i have a matrix for input. Then I want to use transformer to train on. For example use Longformer，i set input_ids to None，and set the input on parameter embed_input. Could it be OK？ anything else need to take care？ I also input the mask.


[BertGeneration] Clean naming

This PR removes all "old" naming of "bert-for-seq-generation" and "BertForSeqGeneration" repectively.


Create model card

Related to pretrained model cards

Create model card for microsoft/layoutlm-large-uncased


Create model card

Related to pretrained model cards

Create model card for microsoft/layoutlm-base-uncased


Further Pretraining of Longformer RAM Consumption

Hello everybody,

I'm interested in continued pretraining of the longformer on in-domain text. I'm using the `run_language_modeling.py` script from the examples with the following options.

!python run_language_modeling.py \
    --output_dir=$output_dir$ \
    --model_type=longformer \
    --model_name_or_path=allenai/longformer-base-4096 \
    --do_train \
    --train_data_file=$train_file$ \
    --save_steps=5000 \
    --save_total_limit=1 \
    --overwrite_output_dir \
    --num_train_epochs=1 \
    --fp16 \
    --fp16_opt_level="O1" \
    --gradient_accumulation_steps=8 \
    --per_device_train_batch_size=1 \
    --logging_steps=100 \
    --line_by_line \
    --mlm

I tried running the script on Google Colab, spell.run and kaggle-notebooks with different behaviour. While on colab and spell.run the consumption of RAM continuously increases while training, it's constant on kaggle. On colab and spell.run the script terminates whenever the limit is reached (around 26gb on spell.run). Since AllenAI provides some code for pretraining as well I tried their implementation as well with the same behaviour, however it reaches RAM limits faster. The text file I'm using is rather small with a few hundred MB in size.

Can someone explain what's going on here and how solve this issue? I'm linking  @patrickvonplaten as it was suggested by the repo. Thanks a lot for any help!

Cheers


Add LayoutLM Model

Related to pretrained model cards

# Introduction
This pull request implements the LayoutLM model, as defined in [the paper](https://arxiv.org/abs/1912.13318):

```
LayoutLM: Pre-training of Text and Layout for Document Image Understanding
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, Ming Zhou, KDD 2020
```

LayoutLM is a simple but effective pre-training method of text and layout for document image understanding and information extraction tasks, such as form understanding and receipt understanding. LayoutLM archives the SOTA results on multiple datasets.

# Typical workflow for including a model

Here an overview of the general workflow:

- [x] Add model/configuration/tokenization classes.
- [ ] Add conversion scripts.
- [x] Add tests and a @slow integration test.
- [ ] Document your model.
- [ ] Finalize.

Let's detail what should be done at each step.

## Adding model/configuration/tokenization classes

Here is the workflow for adding model/configuration/tokenization classes:

- [x] Copy the python files from the present folder to the main folder and rename them, replacing `xxx` with your model
  name.
- [x] Edit the files to replace `XXX` (with various casing) with your model name.
- [x] Copy-paste or create a simple configuration class for your model in the `configuration_...` file.
- [x] Copy-paste or create the code for your model in the `modeling_...` files (PyTorch and TF 2.0).
- [x] Copy-paste or create a tokenizer class for your model in the `tokenization_...` file.

## Adding conversion scripts

Here is the workflow for the conversion scripts:

- [ ] Copy the conversion script (`convert_...`) from the present folder to the main folder.
- [ ] Edit this script to convert your original checkpoint weights to the current pytorch ones.

## Adding tests:

Here is the workflow for the adding tests:

- [x] Copy the python files from the `tests` sub-folder of the present folder to the `tests` subfolder of the main
  folder and rename them, replacing `xxx` with your model name.
- [x] Edit the tests files to replace `XXX` (with various casing) with your model name.
- [x] Edit the tests code as needed.

## Documenting your model:

Here is the workflow for documentation:

- [ ] Make sure all your arguments are properly documented in your configuration and tokenizer.
- [x] Most of the documentation of the models is automatically generated, you just have to make sure that
  `XXX_START_DOCSTRING` contains an introduction to the model you're adding and a link to the original
  article and that `XXX_INPUTS_DOCSTRING` contains all the inputs of your model.
- [x] Create a new page `xxx.rst` in the folder `docs/source/model_doc` and add this file in `docs/source/index.rst`.

Make sure to check you have no sphinx warnings when building the documentation locally and follow our
[documentaiton guide](https://github.com/huggingface/transformers/tree/master/docs#writing-documentation---specification).

## Final steps

You can then finish the addition step by adding imports for your classes in the common files:

- [x] Add import for all the relevant classes in `__init__.py`.
- [x] Add your configuration in `configuration_auto.py`.
- [x] Add your PyTorch and TF 2.0 model respectively in `modeling_auto.py` and `modeling_tf_auto.py`.
- [x] Add your tokenizer in `tokenization_auto.py`.
- [ ] Add a link to your conversion script in the main conversion utility (in `commands/convert.py`)
- [ ] Edit the PyTorch to TF 2.0 conversion script to add your model in the `convert_pytorch_checkpoint_to_tf2.py`
  file.
- [x] Add a mention of your model in the doc: `README.md` and the documentation itself
  in `docs/source/index.rst` and `docs/source/pretrained_models.rst`.
- [x] Upload the pretrained weights, configurations and vocabulary files.
- [ ] Create model card(s) for your models on huggingface.co. For those last two steps, check the
  [model sharing documentation](https://huggingface.co/transformers/model_sharing.html).


EncoderDecoderModel  generate function

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
I follow the EncoderDecoder tutorials(https://huggingface.co/transformers/model_doc/encoderdecoder.html), but it seems that when i use generate function,
generated = model.generate(input_ids, decoder_start_token_id=tokenizer.pad_token_id)
i have to use pad_token_id.If i ommit that parmas, there is a error message, I can't figure out why.

    device=next(self.parameters()).device,
TypeError: full() received an invalid combination of arguments - got (tuple, NoneType, device=torch.device, dtype=torch.dtype), but expected one of:
 * (tuple of ints size, Number fill_value, *, Tensor out, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)
 * (tuple of ints size, Number fill_value, *, tuple of names names, torch.dtype dtype, torch.layout layout, torch.device device, bool pin_memory, bool requires_grad)

tutorials code:
>>> from transformers import EncoderDecoderModel, BertTokenizer
>>> import torch

>>> tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
>>> model = EncoderDecoderModel.from_encoder_decoder_pretrained('bert-base-uncased', 'bert-base-uncased') # initialize Bert2Bert from pre-trained checkpoints

>>> # forward
>>> input_ids = torch.tensor(tokenizer.encode("Hello, my dog is cute", add_special_tokens=True)).unsqueeze(0)  # Batch size 1
>>> outputs = model(input_ids=input_ids, decoder_input_ids=input_ids)

>>> # training
>>> outputs = model(input_ids=input_ids, decoder_input_ids=input_ids, labels=input_ids, return_dict=True)
>>> loss, logits = outputs.loss, outputs.logits

>>> # save and load from pretrained
>>> model.save_pretrained("bert2bert")
>>> model = EncoderDecoderModel.from_pretrained("bert2bert")

>>> # generation
>>> generated = model.generate(input_ids, decoder_start_token_id=model.config.decoder.pad_token_id)

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


circleci testing issue

Sorry, this is not a bug report about `transformers` library. But I don't know what's the best place to ask.

While working on a pull request #6998 , after pushing the code, I got

        ==================================== ERRORS ====================================
        ____________________ ERROR collecting tests/test_trainer.py ____________________
	ImportError while importing test module '/home/circleci/transformers/tests/test_trainer.py'.
	Hint: make sure your test modules/packages have valid Python names.
	Traceback:
	/usr/local/lib/python3.7/importlib/__init__.py:127: in import_module
	    return _bootstrap._gcd_import(name[level:], package, level)
	tests/test_trainer.py:3: in <module>
	    import nlp
	E   ModuleNotFoundError: No module named 'nlp'

The PR #6998  has already been tested several times without problem. And it seems that the error `No module named 'nlp'` is not on my side. Could you help me to resolve this?


Automate the lists in auto-xxx docs

This PR adds a decorate to automate the creation of the lists in the documentation of `AutoConfig`, `AutoTokenizer`, `AutoModel`.

Apart from adding the docstrings of `AutoModelForMultipleChoice` and removing some parts of some class docstrings (because three lists is a bit too much for one class!) this PR does not fix the docstrings in themselves (that's for the next one).


mBART 50

# mBART 50

## Additional languages to the mBART 25 model

The model is described in [Multilingual Translation with Extensible Multilingual Pretraining and Finetuning](https://arxiv.org/pdf/2008.00401.pdf)

## Open source status

* [x] the model implementation is available: [mBART](https://github.com/pytorch/fairseq/tree/master/examples/mbart)
* [ ] the model weights are available: They should be released with the benchmark dataset, but I can't find them.
* [ ] who are the authors: @tangyuq @huihuifan @xianxl ...


these tests require non-multigpu env

fix for https://github.com/huggingface/transformers/issues/7055#issuecomment-690648438

Fixes: #7055

@sgugger


Document the dependcy on datasets

We need the latest version of datasets to run the tests (since Trainer uses it). When datasets is more settled and we don't necessarily need to quickly update it, we can change this to a regular dep in "dev" with a minimal version pinned.


Add option to pass parameters to the loss function.

This PR is by far not finished. In the current state it is just a demo how I would implement it. See #7024

- loss_function_params is a dict that gets passed to the CrossEntropyLoss constructor
- that way you can set class_weights for the `CrossEntropyLoss ` for example
- this is a very important thing to do when working with unbalanced data and can improve your metric by a large amount
- it should be no breaking change since `loss_function_params` is always initialized with `None` which is just the current behavior

Example:
```python
model_name = 'bert-base-german-dbmdz-uncased'

config = AutoConfig.from_pretrained(
    model_name,
    num_labels=3,
)

model = AutoModelForSequenceClassification.from_pretrained(
    model_name,
    config=config,
    loss_function_params={"weight": [0.8, 1.2, 0.97]}
)
```

## ToDo
- [ ] discuss with @LysandreJik @sgugger @nvs-abhilash
- [ ] implement for other models
- [ ] add docstrings
- [ ] write tests
- [ ] maybe implement for TF also


[wip/s2s] DistributedSortishSampler

This allows sortish sampler logic to work on multiple GPU.

The strategy is
1) find the indices that the current rank's data loader should be using (like `DistributedSampler`)
2) reorder those using the `SortishSampler` logic.

### Results
The results on a small MT task are similar to the 1 GPU setting.
+ 2 GPU, random sampler: 13 mins/epoch  BLEU 8.6
+ 2 GPU, `DistributedSortishSampler`: 10 mins/epoch, BLEU 8.6

In the chart below, you can see that the sortish sampler gets to a higher BLEU score in the same number of minutes (x axis) (because it has finished a full epoch rather than 70% of one).

![image](https://user-images.githubusercontent.com/6045025/92787937-87030000-f377-11ea-822a-69e50debe95d.png)

@sgugger let me know if you want this in Trainer!


[testing] test_trainer.py is failing

```
pytest tests/test_trainer.py
```
```
platform linux -- Python 3.7.9, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /mnt/nvme1/code/huggingface/transformers-master
plugins: xdist-2.1.0, forked-1.3.0
collected 11 items

tests/test_trainer.py F.FF.FF...F                                                                                [100%]

====================================================================== FAILURES ======================================================================
____________________________________________________ TrainerIntegrationTest.test_custom_optimizer ____________________________________________________

self = <tests.test_trainer.TrainerIntegrationTest testMethod=test_custom_optimizer>

    def test_custom_optimizer(self):
        train_dataset = RegressionDataset()
        args = TrainingArguments("./regression")
        model = RegressionModel()
        optimizer = torch.optim.SGD(model.parameters(), lr=1.0)
        lr_scheduler = torch.optim.lr_scheduler.LambdaLR(optimizer, lr_lambda=lambda x: 1.0)
        trainer = Trainer(model, args, train_dataset=train_dataset, optimizers=(optimizer, lr_scheduler))
        trainer.train()

>       self.assertTrue(torch.abs(trainer.model.a - 1.8950) < 1e-4)
E       AssertionError: tensor(False, device='cuda:0') is not true

tests/test_trainer.py:240: AssertionError
---------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------
Iteration: 100%|██████████| 4/4 [00:00<00:00,  4.15it/s]
Iteration: 100%|██████████| 4/4 [00:00<00:00, 584.41it/s]
Iteration: 100%|██████████| 4/4 [00:00<00:00, 570.73it/s]
Epoch: 100%|██████████| 3/3 [00:00<00:00,  3.06it/s]
_______________________________________________________ TrainerIntegrationTest.test_model_init _______________________________________________________

self = <tests.test_trainer.TrainerIntegrationTest testMethod=test_model_init>

    def test_model_init(self):
        train_dataset = RegressionDataset()
        args = TrainingArguments("./regression", learning_rate=0.1)
        trainer = Trainer(args=args, train_dataset=train_dataset, model_init=lambda: RegressionModel())
        trainer.train()
>       self.check_trained_model(trainer.model)

tests/test_trainer.py:249:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_trainer.py:105: in check_trained_model
    self.assertTrue(torch.abs(model.a - 0.6975) < 1e-4)
E   AssertionError: tensor(False, device='cuda:0') is not true
---------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------
Iteration: 100%|██████████| 4/4 [00:00<00:00, 540.05it/s]
Iteration: 100%|██████████| 4/4 [00:00<00:00, 510.99it/s]
Iteration: 100%|██████████| 4/4 [00:00<00:00, 553.37it/s]
Epoch: 100%|██████████| 3/3 [00:00<00:00, 130.01it/s]
______________________________________________ TrainerIntegrationTest.test_number_of_steps_in_training _______________________________________________

self = <tests.test_trainer.TrainerIntegrationTest testMethod=test_number_of_steps_in_training>

    def test_number_of_steps_in_training(self):
        # Regular training has n_epochs * len(train_dl) steps
        trainer = get_regression_trainer(learning_rate=0.1)
        train_output = trainer.train()
>       self.assertEqual(train_output.global_step, self.n_epochs * 64 / self.batch_size)
E       AssertionError: 12 != 24.0

tests/test_trainer.py:129: AssertionError
---------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------
Iteration: 100%|██████████| 4/4 [00:00<00:00, 547.43it/s]
Iteration: 100%|██████████| 4/4 [00:00<00:00, 573.03it/s]
Iteration: 100%|██████████| 4/4 [00:00<00:00, 557.12it/s]
Epoch: 100%|██████████| 3/3 [00:00<00:00, 136.18it/s]
_________________________________________________ TrainerIntegrationTest.test_reproducible_training __________________________________________________

self = <tests.test_trainer.TrainerIntegrationTest testMethod=test_reproducible_training>

    def test_reproducible_training(self):
        # Checks that training worked, model trained and seed made a reproducible training.
        trainer = get_regression_trainer(learning_rate=0.1)
        trainer.train()
>       self.check_trained_model(trainer.model)

tests/test_trainer.py:118:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_trainer.py:105: in check_trained_model
    self.assertTrue(torch.abs(model.a - 0.6975) < 1e-4)
E   AssertionError: tensor(False, device='cuda:0') is not true
---------------------------------------------------------------- Captured stderr call ----------------------------------------------------------------
Iteration: 100%|██████████| 4/4 [00:00<00:00, 388.21it/s]
Iteration: 100%|██████████| 4/4 [00:00<00:00, 556.31it/s]
Iteration: 100%|██████████| 4/4 [00:00<00:00, 544.31it/s]
Epoch: 100%|██████████| 3/3 [00:00<00:00, 117.50it/s]
_______________________________________________ TrainerIntegrationTest.test_train_and_eval_dataloaders _______________________________________________

self = <tests.test_trainer.TrainerIntegrationTest testMethod=test_train_and_eval_dataloaders>

    def test_train_and_eval_dataloaders(self):
        trainer = get_regression_trainer(learning_rate=0.1, per_device_train_batch_size=16)
>       self.assertEqual(trainer.get_train_dataloader().batch_size, 16)
E       AssertionError: 32 != 16

tests/test_trainer.py:143: AssertionError
____________________________________________________ TrainerIntegrationTest.test_trainer_with_nlp ____________________________________________________

self = <tests.test_trainer.TrainerIntegrationTest testMethod=test_trainer_with_nlp>

    def test_trainer_with_nlp(self):
        np.random.seed(42)
        x = np.random.normal(size=(64,)).astype(np.float32)
        y = 2.0 * x + 3.0 + np.random.normal(scale=0.1, size=(64,))
        train_dataset = nlp.Dataset.from_dict({"input_x": x, "label": y})

        # Base training. Should have the same results as test_reproducible_training
        model = RegressionModel()
        args = TrainingArguments("./regression", learning_rate=0.1)
>       trainer = Trainer(model, args, train_dataset=train_dataset)

tests/test_trainer.py:212:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/transformers/trainer.py:285: in __init__
    self._remove_unused_columns(self.train_dataset, description="training")
src/transformers/trainer.py:311: in _remove_unused_columns
    dataset.set_format(type=dataset.format["type"], columns=columns)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = Dataset(features: {'input_x': Value(dtype='float32', id=None), 'label': Value(dtype='float64', id=None)}, num_rows: 64), type = 'python'
columns = ['input_x', 'label'], output_all_columns = False, format_kwargs = {}

    def set_format(
        self,
        type: Optional[str] = None,
        columns: Optional[List] = None,
        output_all_columns: bool = False,
        **format_kwargs,
    ):
        """ Set __getitem__ return format (type and columns)

            Args:
                type (Optional ``str``): output type selected in [None, 'numpy', 'torch', 'tensorflow', 'pandas']
                    None means __getitem__ returns python objects (default)
                columns (Optional ``List[str]``): columns to format in the output
                    None means __getitem__ returns all columns (default)
                output_all_columns (``bool`` default to False): keep un-formated columns as well in the output (as python objects)
                format_kwargs: keywords arguments passed to the convert function like `np.array`, `torch.tensor` or `tensorflow.ragged.constant`.
        """
        # Check return type
        if type == "torch":
            try:
                import torch  # noqa: F401
            except ImportError:
                logger.error("PyTorch needs to be installed to be able to return PyTorch tensors.")
        elif type == "tensorflow":
            try:
                import tensorflow  # noqa: F401
            except ImportError:
                logger.error("Tensorflow needs to be installed to be able to return Tensorflow tensors.")
        else:
            assert not (
                type == "pandas" and (output_all_columns or format_kwargs)
            ), "Format type 'pandas' doesn't allow the use of `output_all_columns` or `**format_kwargs`."
            assert (
                type is None or type == "numpy" or type == "pandas"
>           ), "Return type should be None or selected in ['numpy', 'torch', 'tensorflow', 'pandas']."
E           AssertionError: Return type should be None or selected in ['numpy', 'torch', 'tensorflow', 'pandas'].

/home/stas/anaconda3/envs/main-37/lib/python3.7/site-packages/nlp/arrow_dataset.py:542: AssertionError
================================================================== warnings summary ==================================================================
/home/stas/anaconda3/envs/main-37/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py:546
  /home/stas/anaconda3/envs/main-37/lib/python3.7/site-packages/tensorflow/python/data/ops/iterator_ops.py:546: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    class IteratorBase(collections.Iterator, trackable.Trackable,

/home/stas/anaconda3/envs/main-37/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py:106
  /home/stas/anaconda3/envs/main-37/lib/python3.7/site-packages/tensorflow/python/data/ops/dataset_ops.py:106: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    class DatasetV2(collections.Iterable, tracking_base.Trackable,

/home/stas/anaconda3/envs/main-37/lib/python3.7/site-packages/tensorflow/python/autograph/utils/testing.py:21
  /home/stas/anaconda3/envs/main-37/lib/python3.7/site-packages/tensorflow/python/autograph/utils/testing.py:21: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

src/transformers/modeling_tf_utils.py:718
  /mnt/nvme1/code/huggingface/transformers-master/src/transformers/modeling_tf_utils.py:718: DeprecationWarning: invalid escape sequence \s
    """

src/transformers/modeling_funnel.py:130
  /mnt/nvme1/code/huggingface/transformers-master/src/transformers/modeling_funnel.py:130: DeprecationWarning: invalid escape sequence \d
    layer_index = int(re.search("layer_(\d+)", m_name).groups()[0])

tests/test_trainer.py::TrainerIntegrationTest::test_custom_optimizer
tests/test_trainer.py::TrainerIntegrationTest::test_evaluate
tests/test_trainer.py::TrainerIntegrationTest::test_model_init
tests/test_trainer.py::TrainerIntegrationTest::test_number_of_steps_in_training
tests/test_trainer.py::TrainerIntegrationTest::test_predict
tests/test_trainer.py::TrainerIntegrationTest::test_reproducible_training
  /home/stas/anaconda3/envs/main-37/lib/python3.7/site-packages/torch/nn/parallel/_functions.py:61: UserWarning: Was asked to gather along dimension 0, but all input tensors were scalars; will instead unsqueeze and return a vector.
    warnings.warn('Was asked to gather along dimension 0, but all '

-- Docs: https://docs.pytest.org/en/stable/warnings.html
============================================================== short test summary info ===============================================================
FAILED tests/test_trainer.py::TrainerIntegrationTest::test_custom_optimizer - AssertionError: tensor(False, device='cuda:0') is not true
FAILED tests/test_trainer.py::TrainerIntegrationTest::test_model_init - AssertionError: tensor(False, device='cuda:0') is not true
FAILED tests/test_trainer.py::TrainerIntegrationTest::test_number_of_steps_in_training - AssertionError: 12 != 24.0
FAILED tests/test_trainer.py::TrainerIntegrationTest::test_reproducible_training - AssertionError: tensor(False, device='cuda:0') is not true
FAILED tests/test_trainer.py::TrainerIntegrationTest::test_train_and_eval_dataloaders - AssertionError: 32 != 16
FAILED tests/test_trainer.py::TrainerIntegrationTest::test_trainer_with_nlp - AssertionError: Return type should be None or selected in ['numpy', '...
===================================================== 6 failed, 5 passed, 11 warnings in 10.28s ==================================================
```
Env:
```
- `transformers` version: 3.1.0
- Platform: Linux-4.15.0-112-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.9
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: yes
```

Thanks.

 @sgugger?


Fix CI with change of name of nlp

Fixes #7055 (because yes, I can see the future)


[examples] bump pl=0.9.0


TFBert activation layer will be casted into float32 under mixed precision policy

### To reproduce

```
import tensorflow as tf
from transformers import TFBertForQuestionAnswering
tf.keras.mixed_precision.experimental.set_policy('mixed_float16')
model = TFBertForQuestionAnswering.from_pretrained('bert-base-uncased')
```
User will receive warning saying

```
WARNING:tensorflow:Layer activation is casting an input tensor from dtype float16 to the layer's dtype of float32, which is new behavior in TensorFlow 2.  The layer has dtype float32 because its dtype defaults to floatx.

If you intended to run this layer in float32, you can safely ignore this warning. If in doubt, this warning is likely only an issue if you are porting a TensorFlow 1.X model to TensorFlow 2.

To change all layers to have dtype float16 by default, call `tf.keras.backend.set_floatx('float16')`. To change just this layer, pass dtype='float16' to the layer constructor. If you are the author of this layer, you can disable autocasting by passing autocast=False to the base Layer constructor.
```
This log means that under mixed-precision policy, the gelu activation layer in TFBert will be executed with float32 dtype. So the input x([link](https://github.com/huggingface/transformers/blob/d6c08b07a087e83915b4b3156bbf464cebc7b9b5/src/transformers/modeling_tf_bert.py#L91)) will be casted into float32 before sent into activation layer and casted back into float16 after finishing computation.

### Causes
This issue is mainly due to two reasons:

-  The dict of [ACT2FN](https://github.com/huggingface/transformers/blob/d6c08b07a087e83915b4b3156bbf464cebc7b9b5/src/transformers/modeling_tf_bert.py#L121) is defined outside of [TFBertIntermediate class](https://github.com/huggingface/transformers/blob/d6c08b07a087e83915b4b3156bbf464cebc7b9b5/src/transformers/modeling_tf_bert.py#L347). The mixed-precision policy will not be broadcasted into the activation layer defined in this way. This can be verified by adding `print(self.intermediate_act_fn._compute_dtype)` below this [line](https://github.com/huggingface/transformers/blob/d6c08b07a087e83915b4b3156bbf464cebc7b9b5/src/transformers/modeling_tf_bert.py#L360). float32 will be given.

- tf.math.sqrt(2.0) used in [gelu](https://github.com/huggingface/transformers/blob/d6c08b07a087e83915b4b3156bbf464cebc7b9b5/src/transformers/modeling_tf_bert.py#L98) will always return a float32 tensor even if it is under mixed-precision policy. So there will be dtype incompatibility error after we move the ACT2FN into TFBertIntermediate class. This can be further solved by specifying dtype of tf.math.sqrt(2.0) to be consistent with `tf.keras.mixed_precision.experimental.global_policy().compute_dtype
`.

### Pros & Cons
Pros: activation layer will have higher precision as it is computed with float32 dtype
Cons: the two additional Cast operation will increase latency.

### PR
The PR that can help getting rid of warning log and make the activation layer be executed in float16 under mixed-precision policy: https://github.com/jlei2/transformers/pull/3 This PR is not a concrete one because there are some other models importing this ACT2FN dict so it can only solve the issue existed in TFBert related model.

### Benchmark Results
As Huggingface official benchmark tests don't support mixed-precision, the comparison was made using Tensorflow Profiling Tool(https://www.tensorflow.org/tfx/serving/tensorboard).

1 GPU-V100
Model: Huggingface FP16 Bert-Base with batch_size=128 and seq_len = 128.

After applying the PR, the two Cast ops disappear and the time spent on activation layer is reduced from 3.745 ms to 1.75ms.

This issue is not like a bug or error and the pros & cons are listed above. Will you consider make changes corresponding to this?  @patrickvonplaten  Thanks!


How to pass tokenized hypotheses to TFRobertaForSequenceClassification model directly for faster inference?

Hi,
I have been running the zero shot classification pipeline for my use case by passing each text and it’s corresponding list of hypotheses labels, however it takes around 3 hours on a 32B GPU to classify ~22000 sentences(each sentence can have a varying number of labels ranging from 70 to 140 labels).

Is there a way to reduce the computation time by passing the embeddings to the sequence classification model directly rather than the raw list of labels?


[BertGeneration, Docs] Fix another old name in docs


Convert 12-1 and 6-1 en-de models from AllenNLP

https://github.com/jungokasai/deep-shallow#download-trained-deep-shallow-models

+ These should be FSMT models, so can be part of #6940 or done after.
+ They should be uploaded to the AllenNLP namespace. If stas takes this, they can start in stas/ and I will move them.
+ model card(s) should link to the original repo and paper.
+ I hope same en-de tokenizer already ported.
+ Would be interesting to compare BLEU to the initial models in that PR. There is no ensemble so we should be able to reported scores pretty well.
+ **Ideally** this requires 0 lines of checked in python code, besides maybe an integration test.

Desired Signature:
```python
model = FSMT.from_pretrained('allen_nlp/en-de-12-1')
```

Weights can be downloaded with gdown https://pypi.org/project/gdown/

```bash
pip install gdown
gdown https://drive.google.com/uc?id=1x_G2cjvM1nW5hjAB8-vWxRqtQTlmIaQU
```

@stas00 if you are blocked in the late stages of #6940 and have extra cycles, you could give this a whirl. We could also wait for that to be finalized and then either of us can take this.


[BertGeneration] Correct Doc Title

All is in the title.


T5-11b model parallelism

# 🚀 Feature request

I would like to finetune t5-11b model on my dataset, but found that it doesn't fit in TPU or GPU memory - colab notebook just crash when I run it.
I tried to find a ready model parallelism solution. First I found this PR:
https://github.com/huggingface/transformers/pull/3578
but it seems it haven't released. I tried to merge it to master branch locally, and use it, but it's crashed.
Also I have found Eisen library that propose "model parallelism with one code line", but works only for models with only one input ( t5 have 2 inputs - tokens and mask).

I need to distribute model on several GPU, and I see somebody tried to perform it. If this development (pull request 3578) is still in process, can you tell is there are any plans to release it?


Usage of targets argument in fill-mask pipeline (Pipeline cannot handle mixed args and kwargs)

I'm trying to use the `targets`  keyword in the fill mask pipeline as described in #6239 but I'm getting a `ValueError: Pipeline cannot handle mixed args and kwargs`

Full example

    nlp = pipeline('fill-mask', topk=2)
    nlp("The acting was believable and the action was outstanding. The sentiment of this review is <mask>.", targets=[' positive', ' negative'])

    ---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    <ipython-input-44-2f6d875aee9b> in <module>
    ----> 1 nlp("The acting was believable and the action was outstanding. The sentiment of this review is <mask>.",
        2     targets=[' positive', ' negative'])

    ~/miniconda3/envs/huggingface/lib/python3.8/site-packages/transformers/pipelines.py in __call__(self, *args, **kwargs)
    1054
    1055     def __call__(self, *args, **kwargs):
    -> 1056         inputs = self._parse_and_tokenize(*args, **kwargs)
    1057         outputs = self._forward(inputs, return_tensors=True)
    1058

    ~/miniconda3/envs/huggingface/lib/python3.8/site-packages/transformers/pipelines.py in _parse_and_tokenize(self, padding, add_special_tokens, *args, **kwargs)
        503         """
        504         # Parse arguments
    --> 505         inputs = self._args_parser(*args, **kwargs)
        506         inputs = self.tokenizer(
        507             inputs, add_special_tokens=add_special_tokens, return_tensors=self.framework, padding=padding,

    ~/miniconda3/envs/huggingface/lib/python3.8/site-packages/transformers/pipelines.py in __call__(self, *args, **kwargs)
        167     def __call__(self, *args, **kwargs):
        168         if len(kwargs) > 0 and len(args) > 0:
    --> 169             raise ValueError("Pipeline cannot handle mixed args and kwargs")
        170
        171         if len(kwargs) > 0:

    ValueError: Pipeline cannot handle mixed args and kwargs


max_length does not seem to work

So I may have understood the definition wrong, but in my understanding max_length defines the amount of characters you want your summary to be. i.e. max_length=50, the summary is not over 50 characters.

I have installed the transformers, work with the summarizer pipeline, which I assume is the bartsummarizer.
I try to summarize the standard text from the tutorial about the woman who married a lotta guys.
I fill in the following string:
`summarizer = pipeline('summarization')
summarizer(TEXT_TO_SUMMARIZE, min_length=50, max_length=100)`

The summary is this
` Liana Barrientos pleaded not guilty to two counts of "offering a false instrument for filing in the first degree" She has been married to 10 men, nine of them between 1999 and 2002 . At one time, she was married to eight men at once, prosecutors say .`
Which is 249 characters and has weird errors with the . being placed after a space.

It is probably a rookie question, but I can't seem to figure it out.

ps. I run it on Google Colab, if that makes a difference


Small fixes in tf template


Batch_encode_plus with is_pretokenized=True outputs incomplete input_ids

## Environment info

- `transformers` version: 2.11.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1+cu101 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help

 tokenizers: @mfuntowicz

## Information

Model I am using (Bert, XLNet ...): Distilbert, Roberta

I am trying to use batch_encode_plus with pretokenized inputs but the input_encodings are different than if the same text is run individually through encode_plus or if the same text is batch encoded without pretokenization.

## To reproduce

Code and outputs:

```python
tokenizer = AutoTokenizer.from_pretrained('distilbert-base-cased', use_fast=True)
input_text = ['Roman Atwood is a content creator.', 'The Boston Celtics play their home games at TD Garden.']
sample_batch = [x.split(' ') for x in input_text]
sample_batch
```
Output: [['Roman', 'Atwood', 'is', 'a', 'content', 'creator.'],
 ['The', 'Boston',  'Celtics', 'play', 'their', 'home', 'games',  'at', 'TD', 'Garden.']]

```python
encoded_dict_batch = tokenizer.batch_encode_plus(sample_batch, is_pretokenized=True, padding=True, return_tensors='pt', truncation=True, max_length=125)
print(encoded_dict_batch)
```

The output is this which was far fewer non-zero tokens than I expected:

{'input_ids': tensor([[ 101, 2264,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,  102],
        [ 101, 1109,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,    0,
            0,    0,    0,    0,  102]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1],
        [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1]])}

The attention mask is also a problem with all the padded indices being 1s.

Run only the first sentence through encoding (not in batch). All other parameters remain the same.

```python
encoded_dict_single = tokenizer.encode_plus(sample_batch[0], is_pretokenized=True, padding=True, return_tensors='pt', truncation=True, max_length=125)
print(encoded_dict_single)
```
This produces a much more sane output:

{'input_ids': tensor([[ 101, 2264, 1335, 2615, 1110,  170, 3438, 9264,  119,  102]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}

The input ids here from index 2 onwards are replaced by 0 in the previous output even though it's well below max_length.

The words are truncated as well:

```python
print(encoded_dict_single.words())
print(encoded_dict_batch.words())
```
[None, 0, 1, 1, 2, 3, 4, 5, 6, None]
[None, 0, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]

Strangely enough, when calling batch_encode_plus with a pretokenized batch that consists of only 1 item in the list, it works fine.

```python
batch_of_one = [sample_batch[0]]
encoded_dict_batch_of_one = tokenizer.batch_encode_plus(batch_of_one, is_pretokenized=True, padding=True, return_tensors='pt', truncation=True, max_length=125)
print(encoded_dict_batch_of_one)
```
Output:
{'input_ids': tensor([[ 101, 2264, 1335, 2615, 1110,  170, 3438, 9264,  119,  102]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}

I've tried with roberta as well and had the same results. Also, the same sentences in a batch but without pretokenization produces the correct outputs. It seems to be only the combination of pretokenized and batched sentences that are a problem.

## Expected behavior

The input_ids should not be replaced by 0s when tokenizing batched and pre-tokenized inputs.


the time of loading different models to GPU is nearly the same?

I once used the old script for text-classification,the time of loading different models(mode.to_device) to GPU is different,
However, when I run the lastest version of script, I find that loading different models to GPU nearly takes the same time(which is in the process of initing the Trainer)
I want to know the reason.


[wip/token clasification] Introduce datasets and metrics in token classification examples

This PR further simplifies token classification examples by introducing the use of HF nlp datasets and metrics.

In summary, we introduce:

- use of GermEval canonical dataset
- two additional local datasets: [Conll2003](https://github.com/vblagoje/transformers/blob/b58c6186fd589ea6f4c86e10df5a54aa63516d10/examples/token-classification/conll2003.py) and [UDEnglishEWT](https://github.com/vblagoje/transformers/blob/b58c6186fd589ea6f4c86e10df5a54aa63516d10/examples/token-classification/ud_english_ewt.py) (demonstrate custom nlp dataset use in training)
- nlp metrics, splits, removal of seqeval
- removal of all the clutter related to direct download and preprocessing of raw datasets
- further minor simplifications

I have verified the training of all the PyTorch and PL examples works but I was not able to verify the TensorFlow portion due to a faulty GPU setup on my cloud machine. Perhaps @jplu could help me there? Also @stefan-it could you please have a look as well. Therefore, I leave wip tag on, although the PR should be 99% ready.


Fix template


fix to ensure that returned tensors after the tokenization is Long

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #7026


Question about the test results of my own testsets with a fine-tuned BERT

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
Hi, I have a fine-tuned BERT model and I want to use it to predict on my own testset which have 40k sentence pairs. I use the script like:

`export GLUE_DIR=./glue_data
export TASK_NAME=QQP

CUDA_VISIBLE_DEVICES=3 python ./examples/text-classification/run_glue.py \
    --model_name_or_path ./pretrained_weights/bert_base_uncased \
    --task_name $TASK_NAME \
    --do_predict \
    --data_dir $GLUE_DIR/$TASK_NAME \
    --max_seq_length 128 \
    --per_device_eval_batch_size=32   \
    --per_device_train_batch_size=32   \
    --learning_rate 2e-5 \
    --num_train_epochs 3.0 \
    --overwrite_output_dir  \
    --output_dir ./tmp/filter_nouns_pairs \`

It stands to reason that I should get a prediction file containing 40k rows, but in fact the output result is still only 10,000 rows. Is the result truncated during prediction? Thanks!
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


Update eval dataset to pick start_position at first index

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #7032


"You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference" when I am finetuning on distilert pretrained model, After printing this it is taking a lot of time and using only one CPU, how can we parallelized to all the cores in the system ( even I hve 8 GPU's but it is not using tht)

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


add -y to bypass prompt for transformers-cli upload

As discussed here: https://github.com/huggingface/transformers/issues/6934#issuecomment-687365960
adding `-y`/`--yes` to bypass prompt for scripting.

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #6934


[xlm tok] config dict: fix str into int to match definition

match arg doc definition:
```
id2lang (:obj:`Dict[int, str`, `optional`):
```
so replacing str with int.

the inverse is correct.

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #6734


fix deprecation warnings

Fixing:

```
pytest tests/test_tokenization_xlm.py

src/transformers/modeling_tf_utils.py:702
  /mnt/nvme1/code/huggingface/transformers-xlm/src/transformers/modeling_tf_utils.py:702: DeprecationWarning:
invalid escape sequence \s
    """

src/transformers/modeling_funnel.py:130
  /mnt/nvme1/code/huggingface/transformers-xlm/src/transformers/modeling_funnel.py:130: DeprecationWarning:
invalid escape sequence \d
    layer_index = int(re.search("layer_(\d+)", m_name).groups()[0])

tests/test_tokenization_xlm.py::XLMTokenizationTest::test_padding_to_max_length
  /mnt/nvme1/code/huggingface/transformers-xlm/src/transformers/tokenization_utils_base.py:1764: FutureWarning:
The `pad_to_max_length` argument is deprecated and will be removed in a future version, use `padding=True` or `padding='longest'` to pad to the longest sequence in the batch, or use `padding='max_length'` to pad to a max length. In this case, you can give a specific length with `max_length` (e.g. `max_length=45`) or leave max_length to None to pad to the maximal input size of the model (e.g. 512 for Bert).
    warnings.warn(

tests/test_tokenization_xlm.py::XLMTokenizationTest::test_save_and_load_tokenizer
  /mnt/nvme1/code/huggingface/transformers-xlm/src/transformers/tokenization_utils_base.py:1319: FutureWarning:
The `max_len` attribute has been deprecated and will be removed in a future version, use `model_max_length` instead.
    warnings.warn(
```
ok, removing `tests/test_tokenization_common.py`'s `test_padding_to_max_length` as suggested there:
```
    def test_padding_to_max_length(self):
        """We keep this test for backward compatibility but it should be remove when `pad_to_max_length` will e deprecated"""
```
these 2 fail with that test:
```
FAILED tests/test_tokenization_marian.py::MarianTokenizationTest::test_padding_to_max_length
FAILED tests/test_tokenization_pegasus.py::PegasusTokenizationTest::test_padding_to_max_length
```
if I try to fix it:
```
-               padded_sequence_right = tokenizer.encode(sequence, pad_to_max_length=True)
+               padded_sequence_right = tokenizer.encode(sequence, padding="max_length")
```
So there is no salvaging it, right?

Oh and I realized that this one was a `FutureWarning` for end users, but the test suite is under our control, so this is the right action, correct? If I'm wrong, please let me know and I will revert this part.


SQuAD: Implement eval in Trainer-backed run_squad_trainer

A very good first issue IMO!

See https://github.com/huggingface/transformers/pull/4829#issuecomment-645994130

> we should update the eval dataset to pick one start_position (or the most frequent one)

Optionally, use the `huggingface/nlp` library to get the eval dataset, and hook it into the Trainer.

Also referenced in https://github.com/huggingface/transformers/issues/6997#issuecomment-688747680


Unable to recreate onnx speedups demonstrated in 04-onnx-export.ipynb on mac or linux

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.1.0
- Platform: Mac OS Mojave + Ubuntu 18.04.4
- Python version: 3.7.7
- PyTorch version (GPU?): 1.6.0
- Tensorflow version (GPU?): na
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...): bert-base-uncased

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

I am running the /notebooks/04-onnx-export.ipynb example

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

I am using the example data in the notebook
## To reproduce

Steps to reproduce the behavior:

1. Within the notebook add `torch.set_num_threads(1)`
2. Replace `environ["OMP_NUM_THREADS"] = str(cpu_count(logical=True))` with  `environ["OMP_NUM_THREADS"] = "1"`
3. Run the 04-onnx-export.ipynb example notebook

I am trying to recreate the speedups shown in this example notebook.

Note that without step 1 above I found pytorch to be considerably faster than onnx as presumably it was using more threads than onnx, step 2 doesn't seem to impact the results but I set it for completeness (ensuring every thing is on the same number of threads)

Actual results on a Macbook Pro:
![image](https://user-images.githubusercontent.com/53308037/92654284-8ae74180-f2e7-11ea-9397-ac8f8279d7bc.png)

with hardware:
```
machdep.cpu.max_basic: 22
machdep.cpu.max_ext: 2147483656
machdep.cpu.vendor: GenuineIntel
machdep.cpu.brand_string: Intel(R) Core(TM) i7-6700HQ CPU @ 2.60GHz
machdep.cpu.family: 6
machdep.cpu.model: 94
machdep.cpu.extmodel: 5
machdep.cpu.extfamily: 0
machdep.cpu.stepping: 3
machdep.cpu.feature_bits: 9221959987971750911
machdep.cpu.leaf7_feature_bits: 43806655 0
machdep.cpu.leaf7_feature_bits_edx: 2617255424
machdep.cpu.extfeature_bits: 1241984796928
machdep.cpu.signature: 329443
machdep.cpu.brand: 0
machdep.cpu.features: FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE SSE3 PCLMULQDQ DTES64 MON DSCPL VMX EST TM2 SSSE3 FMA CX16 TPR PDCM SSE4.1 SSE4.2 x2APIC MOVBE POPCNT AES PCID XSAVE OSXSAVE SEGLIM64 TSCTMR AVX1.0 RDRAND F16C
machdep.cpu.leaf7_features: RDWRFSGS TSC_THREAD_OFFSET SGX BMI1 HLE AVX2 SMEP BMI2 ERMS INVPCID RTM FPU_CSDS MPX RDSEED ADX SMAP CLFSOPT IPT MDCLEAR TSXFA IBRS STIBP L1DF SSBD
machdep.cpu.extfeatures: SYSCALL XD 1GBPAGE EM64T LAHF LZCNT PREFETCHW RDTSCP TSCI
machdep.cpu.logical_per_package: 16
machdep.cpu.cores_per_package: 8
machdep.cpu.microcode_version: 220
machdep.cpu.processor_flag: 5
machdep.cpu.mwait.linesize_min: 64
machdep.cpu.mwait.linesize_max: 64
machdep.cpu.mwait.extensions: 3
machdep.cpu.mwait.sub_Cstates: 286531872
machdep.cpu.thermal.sensor: 1
machdep.cpu.thermal.dynamic_acceleration: 1
machdep.cpu.thermal.invariant_APIC_timer: 1
machdep.cpu.thermal.thresholds: 2
machdep.cpu.thermal.ACNT_MCNT: 1
machdep.cpu.thermal.core_power_limits: 1
machdep.cpu.thermal.fine_grain_clock_mod: 1
machdep.cpu.thermal.package_thermal_intr: 1
machdep.cpu.thermal.hardware_feedback: 0
machdep.cpu.thermal.energy_policy: 1
machdep.cpu.xsave.extended_state: 31 832 1088 0
machdep.cpu.xsave.extended_state1: 15 832 256 0
machdep.cpu.arch_perf.version: 4
machdep.cpu.arch_perf.number: 4
machdep.cpu.arch_perf.width: 48
machdep.cpu.arch_perf.events_number: 7
machdep.cpu.arch_perf.events: 0
machdep.cpu.arch_perf.fixed_number: 3
machdep.cpu.arch_perf.fixed_width: 48
machdep.cpu.cache.linesize: 64
machdep.cpu.cache.L2_associativity: 4
machdep.cpu.cache.size: 256
machdep.cpu.tlb.inst.large: 8
machdep.cpu.tlb.data.small: 64
machdep.cpu.tlb.data.small_level1: 64
machdep.cpu.address_bits.physical: 39
machdep.cpu.address_bits.virtual: 48
machdep.cpu.core_count: 4
machdep.cpu.thread_count: 8
machdep.cpu.tsc_ccc.numerator: 216
machdep.cpu.tsc_ccc.denominator: 2
```

I obtained even worse results on a linux machine:
![image](https://user-images.githubusercontent.com/53308037/92656112-650f6c00-f2ea-11ea-9021-1419ec20ba97.png)

with hardware:
```
processor       : 11
vendor_id       : GenuineIntel
cpu family      : 6
model           : 63
model name      : Intel(R) Core(TM) i7-5820K CPU @ 3.30GHz
stepping        : 2
microcode       : 0x43
cpu MHz         : 1199.433
cache size      : 15360 KB
physical id     : 0
siblings        : 12
core id         : 5
cpu cores       : 6
apicid          : 11
initial apicid  : 11
fpu             : yes
fpu_exception   : yes
cpuid level     : 15
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts md_clear flush_l1d
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs itlb_multihit
bogomips        : 6596.76
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
Expected to speed speedup from using onnx as in the example:
![image](https://user-images.githubusercontent.com/53308037/92653635-8a01e000-f2e6-11ea-9df2-76ac8879dd3f.png)

I know this is hardware specific but having tested it on two machines I wonder if there is some config not included in the example that I am missing or some other issue?


[s2s] dynamic batch size with --max_tokens_per_batch

+ adds test coverage for `SortishSampler`+`DistributedSortishSampler`.
+ use fancy samplers for validation if specified.
+ dynamic batch sampler is still experimental. Distributed doesn't work, only slightly faster at the moment, and requires a preprocessing step.


Add TF Funnel Transformer

This adds the TF implementation of the model. Will upload the TF checkpoints as the PR goes under review.


No way around "Truncation was not explicitely activated..." error when using SingleSentenceClassificationProcessor.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.1.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?:

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->
@LysandreJik, @thomwolf

## Information

Model I am using: BERT.

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

```
from transformers import *
processor = SingleSentenceClassificationProcessor()
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
processor.add_examples(["Thanks for cool stuff!"])
processor.get_features(tokenizer, max_length=3)
```
```
Truncation was not explicitely activated but `max_length` is provided a specific value, please use `truncation=True` to explicitely truncate examples to max length. Defaulting to 'longest_first' truncation strategy. If you encode pairs of sequences (GLUE-style) with the tokenizer you can select this strategy more precisely by providing a specific strategy to `truncation`.

[InputFeatures(input_ids=[101, 4283, 102], attention_mask=[1, 1, 1], token_type_ids=None, label=0)]
```
## Expected behavior
This is expected, but the problem is that there is no way to suppress that warning, because there is no way to pass `truncation=True` when `tokenizer.encode` is called within `processor.get_features`. Probably one should make `truncation` an argument to `processor.get_features`.


Getting underling S3 URL

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
The high performance cluster environment at my work has been great for working with transformers. The downside however is that most access to the internet has been locked down. This means that I have been manually bringing in the models. This is somewhat problematic since it is a laborious process and I have been behind on being able to update to the newest models.

The IT team has however said that they would be able to white list a specific S3 bucket. When I see the link to the models, I see that it is of the form: https://s3.amazonaws.com/models.huggingface.co rather than the usual https://bucket-name.s3.Region.amazonaws.com.

Is there any chance I could obtain that second format for the bucket?


RuntimeError: Expected tensor for argument #1 'indices' to have scalar type Long; but got torch.FloatTensor instead (while checking arguments for embedding)

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.1.0
- Platform: Linux-4.19.0-10-amd64-x86_64-with-debian-10.5
- Python version: 3.6.10
- PyTorch version (GPU?): 1.3.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger

 -->

## Information

Model I am using (GPT2-large) for fine-tuning on custom data:

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

Trace:

>   File "gpt_language_generation.py", line 209, in <module>
>     main()
>   File "gpt_language_generation.py", line 136, in main
>     trainer.train(model_path=None)
>   File "<conda_env>/lib/python3.6/site-packages/transformers/trainer.py", line 708, in train
>     tr_loss += self.training_step(model, inputs)
>   File "<conda_env>/lib/python3.6/site-packages/transformers/trainer.py", line 995, in training_step
>     outputs = model(**inputs)
>   File "<conda_env>/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
>     result = self.forward(*input, **kwargs)
>   File "<conda_env>/lib/python3.6/site-packages/transformers/modeling_gpt2.py", line 731, in forward
>     return_dict=return_dict,
>   File "<conda_env>/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
>     result = self.forward(*input, **kwargs)
>   File "<conda_env>/lib/python3.6/site-packages/transformers/modeling_gpt2.py", line 593, in forward
>     inputs_embeds = self.wte(input_ids)
>   File "<conda_env>/lib/python3.6/site-packages/torch/nn/modules/module.py", line 541, in __call__
>     result = self.forward(*input, **kwargs)
>   File "<conda_env>/lib/python3.6/site-packages/torch/nn/modules/sparse.py", line 114, in forward
>     self.norm_type, self.scale_grad_by_freq, self.sparse)
>   File "<conda_env>/lib/python3.6/site-packages/torch/nn/functional.py", line 1484, in embedding
>     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:
```
model_class, tokenizer_class = GPT2LMHeadModel, GPT2Tokenizer
model = model_class.from_pretrained("gpt2-large")
tokenizer = tokenizer_class.from_pretrained("gpt2-large")
special_tokens_dict = {'bos_token': '<BOS>', 'eos_token': '<EOS>', 'pad_token': '<PAD>'}
num_added_toks = tokenizer.add_special_tokens(special_tokens_dict)
model.resize_token_embeddings(len(tokenizer))
input_text = ["a cat is sitting on the mat"]*100
train_dataset = tokenizer(input_text,add_special_tokens=True, truncation=True, max_length=64)
train_dataset = train_dataset["input_ids"]
eval_dataset = tokenizer(input_text,add_special_tokens=True, truncation=True, max_length=64)
eval_dataset = eval_dataset["input_ids"]
data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer,mlm=False)
training_args = TrainingArguments(output_dir='./gpt_model/')
training_args.do_train = True
training_args.do_eval = True
training_args.per_device_train_batch_size = 32
trainer = Trainer(
           model=model,
            args=training_args,
            data_collator=data_collator,
            train_dataset=train_dataset,
            eval_dataset=eval_dataset,
            prediction_loss_only=True)

trainer.train(model_path=None) // The error occurs here
trainer.save_model()
```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

Expect the training to continue without an error.


Python


Adding `class_weights` argument for the loss function of transformers model

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

To provide a parameter called `class_weights` while initializing a sequence classification model. The attribute will be used to calculate the weighted loss which is useful for classification with imbalanced datasets.

```python
from transformers import DistilBertForSequenceClassification

# Note the additional class_weights attribute
model = DistilBertForSequenceClassification.from_pretrained(
    "distilbert-base-uncased",
    num_labels=5,
    class_weights=[5, 3, 2, 1, 1])
```

class_weights will provide the same functionality as the `weight` parameter of Pytorch losses like [torch.nn.CrossEntropyLoss](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html).

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

There have been similar issues raised before on "How to provide class weights for imbalanced classification dataset". See [#297](https://github.com/huggingface/transformers/issues/297#issuecomment-534185049), [#1755](https://github.com/huggingface/transformers/issues/1755),

And I ended up modifying the transformers code to get the class weights (shown below), and it looks like an easy addition which can benefit many.

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

This should be possible because currently the loss for Sequence classification in the forward method is initialized like below:
```python
    loss_fct = nn.CrossEntropyLoss() # <- Defined without the weight parameter
    loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
```

And we can add the weight attribute of Pytorch and pass the class_weights recorded during model initialization.
```python
    loss_fct = nn.CrossEntropyLoss(weight=self.class_weights)
    loss = loss_fct(logits.view(-1, self.num_labels), labels.view(-1))
```

I am happy to implement this and provide a PR. Although I am new to the transformers package and may require some iterative code reviews from the senior contributors/members.


PreTrained (custom) model not correctly initializing when using AutoModel methods

## Environment info

- `transformers` version: 3.1.0
- Platform: Windows-10-10.0.18362-SP0
- Python version: 3.8.5
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: ???

### Who can help
@LysandreJik

## Information

Model I am using (Bert, XLNet ...): bert-base-uncased, roberta-base, albert-large-v2

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [X] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [X] my own task or dataset: (give details below)

## To reproduce

(Conceptual) steps to reproduce the behavior:

1) Derive a custom model from a "PreTrainedModel" like:

class TransformerLSTMDocClassifier(PreTrainedModel):

    def __init__(self, model_config: AutoConfig):

        super(TransformerLSTMDocClassifier, self).__init__(model_config)
        self.transformer = AutoModel.from_config(model_config)
        self.dropout = nn.Dropout(p=model_config.hidden_dropout_prob)
        self.lstm = LSTM(model_config.hidden_size, model_config.hidden_size)
        self.classifier = nn.Sequential(
                        nn.Dropout(p=model_config.hidden_dropout_prob),
                        nn.Linear(model_config.hidden_size, model_config.num_labels),
                        nn.Tanh()
                    )

2) Train (fine-tune) the custom model starting from a pre-trained (standard) model such as bert-base-uncased, roberta-base, albert-large-v2 (using AutoModel features). My objective is to easily exchange the 'transformer' part in the above model

3) Initializing the model class 'TransformerLSTMDocClassifier' via the 'from_pretrained' method with the fine-tuned model (output of step 2) results in the message:

Some weights of TransformerLSTMDocClassifier were not initialized from the model checkpoint at <MODEL_CHECKPOINT> and are newly initialized: [...]

The list of weights apparently includes ALL weights of the TransformerLSTMDocClassifier  (200+in case of using bert-base_uncased)

## Expected behavior

Properly initialized weights

## Addendum

Disclaimer: I am a novice in the "transformers" framework. The above described (unexpected) behavior might result from an incorrect use of 'transformers' features/methods at my end.

However, I was digging a bit in the code and made the following observations:

(1) in case of deriving the custom model from a specific class (e.g. BertPreTrainedModel in case of using bert-base-uncased) instead from the generic class 'PreTrainedModel', the custom model is correctly initialized.

(2) the 'cls.base_model_prefix' (module: modeling_utils; method: from_pretrained) is "" (empty string) in case the custom model is derived from "PreTrainedModel", resulting in 'has_prefix_module' being set to 'True' in line 925, finally resulting in an inappropriate 'start_prefix' (apparently preventing that weights are matched/loaded).


New TF output proposal

Hello!

Currently the TensorFlow have several issues to properly working in graph mode when `output_attentions`, `outout_hidden_states` and `return_dict` takes a boolean tensor (`tf.constant(True/False)`). This is because the graph mode doesn't allows to have an output of different sizes in its conditional branches.

To fix this and lower as much as possible a breaking change with the current behavior of these models, I propose to keep the current behavior but only in eager mode, and disabling these three features (force to have them to False) in graph mode. This is for me the best compromise between having something that works in graph mode and not having a big breaking change.

The graph mode is most of the time used to get fast training/inference and not for doing experiments, and I don't see the point to deploy a model in production that gives you all the attentions/hidden states values, or possibly getting an OOM during the training/inference mostly for the new TFLongformer model.

Here some examples.

Example 1: run in eager mode with default parameters value
```python
from transformers import TFBertForMaskedLM

model = TFBertForMaskedLM.from_pretrained("bert-base-cased")
model(tf.constant([[10,11,12]]))

# Result
(<tf.Tensor: shape=(1, 3, 28996), dtype=float32, numpy=
array([[[ -9.900146 , -10.345711 ,  -9.721893 , ...,  -7.7057724,
          -8.035212 ,  -8.271875 ],
        [ -7.866758 ,  -8.147887 ,  -7.864182 , ...,  -6.7170696,
          -6.2896423,  -6.79779  ],
        [ -9.790001 , -10.2271185,  -9.584716 , ...,  -7.539982 ,
          -7.807548 ,  -8.135937 ]]], dtype=float32)>,)
```

Example 2: run in eager mode by updating a boolean parameter value
```python
from transformers import TFBertForMaskedLM

model = TFBertForMaskedLM.from_pretrained("bert-base-cased")
model(tf.constant([[10,11,12]]), return_dict=True)

# Result
TFMaskedLMOutput(loss=None, logits=<tf.Tensor: shape=(1, 3, 28996), dtype=float32, numpy=
array([[[ -9.900146 , -10.345711 ,  -9.721893 , ...,  -7.7057724,
          -8.035212 ,  -8.271875 ],
        [ -7.866758 ,  -8.147887 ,  -7.864182 , ...,  -6.7170696,
          -6.2896423,  -6.79779  ],
        [ -9.790001 , -10.2271185,  -9.584716 , ...,  -7.539982 ,
          -7.807548 ,  -8.135937 ]]], dtype=float32)>, hidden_states=None, attentions=None)
```

Example 3: run in graph mode with default parameters value
```python
from transformers import TFBertForMaskedLM
import tensorflow as tf

model = tf.function(TFBertForMaskedLM.from_pretrained("bert-base-cased"))
model(tf.constant([[10,11,12]]))

# Result
(<tf.Tensor: shape=(1, 3, 28996), dtype=float32, numpy=
array([[[ -9.900146 , -10.345711 ,  -9.721893 , ...,  -7.7057724,
          -8.035212 ,  -8.271875 ],
        [ -7.866758 ,  -8.147887 ,  -7.864182 , ...,  -6.7170696,
          -6.2896423,  -6.79779  ],
        [ -9.790001 , -10.2271185,  -9.584716 , ...,  -7.539982 ,
          -7.807548 ,  -8.135937 ]]], dtype=float32)>,)
```

Example 4: run in graph mode by updating a boolean parameter value
```python
from transformers import TFBertForMaskedLM
import tensorflow as tf

model = tf.function(TFBertForMaskedLM.from_pretrained("bert-base-cased"))
model(tf.constant([[10,11,12]]), return_dict=tf.constant(True))

# Result
(<tf.Tensor: shape=(1, 3, 28996), dtype=float32, numpy=
array([[[ -9.900146 , -10.345711 ,  -9.721893 , ...,  -7.7057724,
          -8.035212 ,  -8.271875 ],
        [ -7.866758 ,  -8.147887 ,  -7.864182 , ...,  -6.7170696,
          -6.2896423,  -6.79779  ],
        [ -9.790001 , -10.2271185,  -9.584716 , ...,  -7.539982 ,
          -7.807548 ,  -8.135937 ]]], dtype=float32)>,)
```
As you can see for the last example, no more graph compilation error due to the `tf.constant(True)` value.

For my examples I used only `return_dict` but of course the same behavior is applied to `output_attentions` and `output_hidden_states`. Moreover, this new behavior will be detailed in the documentation of the impacted parameters.

@LysandreJik @sgugger @patrickvonplaten @sshleifer @thomwolf @julien-c What do you think of this solution?


where can I download the pre-trained pytorch_model.bin files ?


# ❓ Questions & Help
Use  `run_language_modeling.py` to finetune gpt2, but it core unexpectedly. How can i find the cause of the core please?

## Details

I'm using transformers's `run_language_modeling.py` to finetune gpt2  as follows.

### （1）with gpu
```shell
python3 run_language_modeling.py     --output_dir=/home/xxx/transformers/examples/language-modeling/output_dir     --model_type=gpt2     --model_name_or_path=gpt2     --per_gpu_train_batch_size=1     --do_train     --train_data_file=/home/xxx/data_info/transformer.data     --block_size=512 --save_steps=500 --overwrite_output_dir
```
But it core unexpectedly.
```shell
terminate called after throwing an instance of 'std::runtime_error'                                                                                | 0/1348 [00:00<?, ?it/s]
  what():  NCCL Error 1: unhandled cuda error
Aborted
```
And the returncode is `134`。
![image](https://user-images.githubusercontent.com/12140508/92548004-863f7080-f288-11ea-8e26-50532f59c516.png)

### (2) with cpu
```shell
python3 run_language_modeling.py     --output_dir=/home/xxx/transformers/examples/language-modeling/output_dir     --model_type=gpt2     --model_name_or_path=gpt2        --do_train     --train_data_file=/home/xxx/data_info/transformer.data     --block_size=512 --save_steps=500 --overwrite_output_dir --no_cuda
```
At first the training is normal , but will quit after a while with return code `137`.
![image](https://user-images.githubusercontent.com/12140508/92548295-39a86500-f289-11ea-86c1-d91d13121162.png)

### (3) question
The dataset i use is just a `data.txt` which is a file combined with  multiple articles ,  and a  `<|endoftext|>` is added at the end of each article. How can i find the cause of the core please? Hope for your help. Thanks a lot.


Proposal: Offset based Token Classification utilities

# 🚀 Feature request

Hi. So we work a lot with span annotations on text that isn't tokenized and want a "canonical" way to work with that. I have some ideas and rough implementations, so I'm looking for feedback on if this belongs in the library, and if the proposed implementation is more or less good.

I also think there is a good chance that everything I want exists, and the only solution needed is slightly clearer documentation. I should hope that's the case and happy to document if someone can point me in the right direction.

## The Desired Capabilities

What I'd like is a canonical way to:
* Tokenize the examples in the dataset
* Align my annotations with the output tokens (see notes below)
* Have the tokens and labels correctly padded to the max length of an example in the batch or max_sequence_length
* Have a convenient function that returns predicted offsets

### Some Nice To Haves
* It would be nice if such a utility internally handled tagging schemes like IOB BIOES internally and optionally exposed them in the output or "folded" them to the core entities.
* It would be nice if there was a recommended/default strategy implemented for handling examples that are longer then the max_sequence_length
* It would be amazing if we could pass labels to the tokenizer and have the alignment happen in Rust (in parallel). But I don't know Rust and I have a sense this is complicated so I won't be taking that on myself, and assuming that this is happening in Python.

## Current State and what I'm missing

* The docs and examples for Token Classification assume that the [text is pre-tokenized](https://github.com/huggingface/transformers/blob/1b76936d1a9d01cf99a086a3718060a64329afaa/examples/token-classification/utils_ner.py#L110).
* For a word that has a label and is tokenized to multiple tokens, [it is recommended](https://github.com/huggingface/transformers/blob/1b76936d1a9d01cf99a086a3718060a64329afaa/examples/token-classification/utils_ner.py#L116)  to place the label on the first token and "ignore" the following tokens
   * However it is not clear where that recommendation came from, and it has [edge cases that seem quite nasty](https://github.com/huggingface/transformers/issues/5077#issuecomment-668384357)
* The [example pads all examples to max_sequence_length](https://github.com/huggingface/transformers/blob/1b76936d1a9d01cf99a086a3718060a64329afaa/examples/token-classification/utils_ner.py#L167)  which is a big performance hit (as opposed to bucketing by length and padding dynamically)
* The example loads the entire dataset at once in memory. I'm not sure if this is a real problem or I'm being nitpicky, but I think "the right way" to do this would be to lazy load a batch or a few batches.

### Alignment
The path to align tokens to span annotations  is by using the return_offsets_mapping flag on the tokenizer (which is awesome!).
There are probably a few strategies, I've been using this
I use logic like this:
```python
def align_tokens_to_annos(offsets,annos):
    anno_ix =0
    results =[]
    done =len(annos)==0
    for offset in offsets:

        if done == True:
            results.append(dict(offset=offset,tag='O',))
        else:
            anno = annos[anno_ix]
            start, end = offset
            if end < anno['start']:
                # the offset is before the next annotation
                results.append(dict(offset=offset, tag='O', ))
            elif start <=anno['start'] and end <=anno['end']:
                results.append(dict(offset=offset, tag=f'B-{anno["tag"]}',))
            elif start>=anno['start'] and end<=anno['end']:
                results.append(dict(offset=offset, tag=f'I-{anno["tag"]}', ))
            elif start>=anno['start'] and end>anno['end']:
                anno_ix += 1
                results.append(dict(offset=offset, tag=f'E-{anno["tag"]}', ))
            else:
                raise Exception(f"Funny Overlap {offset},{anno}",)

            if anno_ix>=len(annos):
                done=True
    return results
```
 And then call that function inside add_labels here
```python
        res_batch = tokenizer([s['text'] for s in pre_batch],return_offsets_mapping=True,padding=True)
        offsets_batch = res_batch.pop('offset_mapping')
        res_batch['labels'] =[]
        for i in range(len(offsets_batch)):
          labels = add_labels(res_batch['input_ids'][i],offsets_batch[i],pre_batch[i]['annotations'])
          res_batch['labels'].append(labels)
````

This works, and it's nice because the padding is consistent with the longest sentence so bucketing gives a big boost. But, the add_labels stuff is in python and thus sequential over the examples and not super fast. I haven't measured this to confirm it's a problem, just bring it up.

## Desired Solution

I need most of this stuff so I'm going to make it. I could do it

The current "NER" examples and issues assume that text is pre-tokenized. Our use case is such that the full text is not tokenized and the labels for "NER" come as offsets. I propose a utility /example to handle that scenario because I haven't been able to find one.

In practice, most values of X don't need any modification, and doing what I propose (below) in Rust is beyond me, so this might boil down to a utility class and documentation.

## Motivation

I make [text annotation tools](https://lighttag.io) and our output is span annotations on untokenized text. I want our users to be able to easily use transformers. I suspect from my (limited) experience that in many non-academic use cases, span annotations on untokenized text is the norm and that others would benefit from this as well.

## Possible ways to address this

I can imagine a few scenarios here

 * **This is out of scope** Maybe this isn't something that should be handled by transformers at all, and delegated to a library and blog post
* **This is in scope and just needs documentation** e.g. all the things I mentioned are things transformers should and can already do. In that case the solution would be pointing someone (me) to the right functions and adding some documentation
* **This is in scope and should be a set of utilities ** Solving this could be as simple as making a file similar to [utils_ner.py](https://github.com/huggingface/transformers/blob/1b76936d1a9d01cf99a086a3718060a64329afaa/examples/token-classification/utils_ner.py). I think that would be the simplest way to get something usable and gather feedback see if anyone else cares
* **This is in scope but should be done in Rust soon** If we want to be performance purists, it would make sense to handle the alignment of span based labels in Rust. I don't know Rust so I can't help much and I don't know if there is any appetite or capacity from someone that does, or if it's worth the (presumably) additional effort.

## Your contribution

I'd be happy to implement and submit a PR, or make an external library or add to a relevant existing one.

## Related issues
* #5297
* [This PR](https://github.com/huggingface/transformers/pull/3957)


[s2s] --eval_max_generate_length


pegasus.rst: fix expected output

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #{issue number}


[Longformer] Fix longformer documentation

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #7015


Longformer global attention mask, 2 or 1?

This example in the LongFormer documentation (https://huggingface.co/transformers/model_doc/longformer.html )
```
attention_mask[:, [1, 4, 21,]] = 2  # Set global attention based on the task. For example,
...                                     # classification: the <s> token
...                                     # QA: question tokens
...                                     # LM: potentially on the beginning of sentences and paragraphs
```
seems to mislead the user to put a 2 as the global attention mask instead of a 1 this is described in the actual documentation. I'm guessing this is a mistake because (a) the example is directly copied from the AllenAI (https://github.com/allenai/longformer#how-to-use), and (b) the source code actually adds a +1 to the attention mask in the function `_merge_to_attention_mask` (https://huggingface.co/transformers/_modules/transformers/modeling_longformer.html#LongformerModel.forward).

Could the documentation please be fixed or clarified? Thanks!

Also what is the 1, 4, 21 in that example?

### Who can help
 Longformer/Reformer: @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): LongFormer


[wip] Pegasus: Hack to never generate unk


Fixing FLOPS merge by checking if torch is available