pip install error

Hi, I tried to install transformers library via `pip install transformers` and I got tokenizer install error.

The error logs are as follows.
```
ERROR: Could not find a version that satisfies the requirement tokenizers==0.8.1.rc1 (from transformers) (from versions: 0.0.2, 0.0.3, 0.0.4, 0.0.5, 0.0.6, 0.0.7, 0.0.8, 0.0.9, 0.0.10, 0.0.11, 0.0.12, 0.0.13, 0.1.0, 0.1.1, 0.2.0, 0.2.1, 0.3.0, 0.4.0, 0.4.1, 0.4.2, 0.5.0, 0.5.1, 0.5.2, 0.6.0, 0.7.0, 0.8.0)
ERROR: No matching distribution found for tokenizers==0.8.1.rc1 (from transformers)
```

I googled about it but I couldn't find the way to solve it.
Is there I can do to handle this issue?

Thank you.


Update shortcut name for reformer in pretrained_models.srt

Adding `google/` prefix on shortcut name for reformer in `pretrained_models.srt` according to mapping in `configuration_reformer.py`


Hi,

After changing from `BERT_PRETRAINED_MODEL_ARCHIVE_MAP` to `BERT_PRETRAINED_MODEL_ARCHIVE_LIST` in #5842 , I encountered another issue:

>   File "main.py", line 24, in <genexpr>
>     ALL_MODELS = sum((tuple(conf.pretrained_config_archive_map.keys()) for conf in (BertConfig, XLNetConfig)), ())
> AttributeError: type object 'BertConfig' has no attribute 'pretrained_config_archive_map'

Is it also a breaking change ?
What is the replacing name for `pretrained_config_archive_map` now ?

Thank you,


Create README.md

model card

Related to pretrained model cards


Decode [UNK] from tokenizer

I am using Bert for NER and I have a problem when the tokenizer uses the token [UNK] to mask a word. How can I easily know which was the original work given its index?


Added model card for neuraly/bert-base-italian-cased-sentiment

Related to pretrained model cards

Hello, we are making this pull request to add our Italian sentiment model to your repository of transformers.
Thank you again for hosting the models :)


problem about Custom class inheriting from <TFBertPreTrainedModel>

# ❓ Questions & Help

## Details
<!-- Description of your issue -->
I constructed class inheriting from TFBertPreTrainedModel. as follows:
```
class Model(transformers.TFBertPreTrainedModel):
    def __init__(self,config, *inputs, **kwargs):
        super(Model,self).__init__(config, *inputs, **kwargs)
        self.bert = transformers.TFBertMainLayer(config,name="bert")

    @tf.function
    def call(self, inputs, training=None, mask=None,**kwargs):
        out = self.bert(inputs)
        print(out)
        return out
```
main progress as follow:
```
if __name__ == "__main__":
    tokenizer = transformers.BertTokenizer("model/chinese_L-12_H-768_A-12/vocab.txt")
    text_2 = tokenizer.batch_encode_plus(["你买啊，买了就是成都人", "你来啊，来了就是深圳人"], max_length=20, pad_to_max_length=True)
    print(text_2)
    model = Model.from_pretrained("bert-base-chinese")
    out = model([tf.convert_to_tensor(text_2["input_ids"]),tf.convert_to_tensor(text_2['attention_mask'])])
```

unfortunately，the print on console was different for what I expect when I run this program. I expect that the out of **print(out)** is a tensor with size [2,20,768], but in fact, the print was run three times, outputs is **(<tf.Tensor 'bert/encoder/layer_._11/output/LayerNorm/batchnorm/add_1:0' shape=(3, 5, 768) dtype=float32>, <tf.Tensor 'bert/pooler/dense/Tanh:0' shape=(3, 768) dtype=float32>)** , **(<tf.Tensor 'bert/encoder/layer_._11/output/LayerNorm/batchnorm/add_1:0' shape=(3, 5, 768) dtype=float32>, <tf.Tensor 'bert/pooler/dense/Tanh:0' shape=(3, 768) dtype=float32>)** and **(<tf.Tensor 'bert/encoder/layer_._11/output/LayerNorm/batchnorm/add_1:0' shape=(2, 20, 768) dtype=float32>, <tf.Tensor 'bert/pooler/dense/Tanh:0' shape=(2, 768) dtype=float32>)**
My question are 1) why 2)how could I get the last outputs for following layers
I am looking forward your help, thanks
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


Related to pretrained model cards


Hi,

I'm rerunning the code BERT-E2E-ABSA from https://github.com/lixin4ever/BERT-E2E-ABSA, which is based on transformers 2.0.0

I installed the latest transformers 3.0.2 (both from pip and souce) and I have an error that I can not import BERT_PRETRAINED_MODEL_ARCHIVE_MAP.
This simple command will result in errors below:
> python -c "from transformers import BERT_PRETRAINED_MODEL_ARCHIVE_MAP"

> Traceback (most recent call last):
>   File "<string>", line 1, in <module>
> ImportError: cannot import name 'BERT_PRETRAINED_MODEL_ARCHIVE_MAP' from 'transformers' (/home/hoa/transformers/src/transformers/__init__.py)

Transformer is well installed as it can do the following:
> python -c "from transformers import pipeline; print(pipeline('sentiment-analysis')('I hate you'))"
> [{'label': 'NEGATIVE', 'score': 0.9991129040718079}]

Do you know why ? Did I miss something ?

- `transformers` version: 3.0.2
- Platform: Linux-5.4.0-1019-gcp-x86_64-with-debian-bullseye-sid
- Python version: 3.7.7
- PyTorch version (GPU?): 1.2.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No
- Other configurations: Ubuntu 20.04, Python 3.7, CUDA 11

Note: I even installed pytorch on cpu laptop and rerun, still had the same issue !


[Model card] Bert2Bert

Related to pretrained model cards


[WIP - Don't merge][EncoderDecoder] Extend Trainer for Bert2Bert

This PR draft should be used to train a Bert2Bert model.
It's not ready to be merged yet.


Created model card for my extreme summarization model

Related to pretrained model cards


Created model card for my summarization model

Related to pretrained model cards


[seq2seq] MAX_LEN env var for MT commands


[not sure whether to] pin torch<=1.5.1


solve Illegal seek in wandb teardown (test suite)

somehow wandb teardown fails in `tests/test_trainer.py::TrainerIntegrationTest::test_trainer_eval_mrpc`:

```
pytest -n 1 tests/test_trainer.py::TrainerIntegrationTest::test_trainer_eval_mrpc
====================================================================== test session starts =======================================================================
platform linux -- Python 3.7.5, pytest-5.4.3, py-1.9.0, pluggy-0.13.1
rootdir: /mnt/nvme1/code/huggingface/transformers-tests-1
plugins: hypothesis-5.5.4, filter-subpackage-0.1.1, arraydiff-0.3, flaky-3.6.1, ipynb-1.1.1.dev0, cov-2.10.0, astropy-header-0.1.2, forked-1.2.0, doctestplus-0.5.0, openfiles-0.4.0, remotedata-0.3.2, xdist-1.32.0
gw0 [1]
FE                                                                                [100%]
wandb: Waiting for W&B process to finish, PID 19525

============================================================================= ERRORS =============================================================================
_______________________________________________ ERROR at teardown of TrainerIntegrationTest.test_trainer_eval_mrpc _______________________________________________
[gw0] linux -- Python 3.7.5 /home/stas/anaconda3/envs/main/bin/python

self = <contextlib._GeneratorContextManager object at 0x7efc47610690>, type = None, value = None, traceback = None

    def __exit__(self, type, value, traceback):
        if type is None:
            try:
>               next(self.gen)
E               OSError: [Errno 29] Illegal seek

/home/stas/anaconda3/envs/main/lib/python3.7/contextlib.py:119: OSError
============================================================================ FAILURES ============================================================================
_________________________________________________________ TrainerIntegrationTest.test_trainer_eval_mrpc __________________________________________________________
[gw0] linux -- Python 3.7.5 /home/stas/anaconda3/envs/main/bin/python

self = <contextlib._GeneratorContextManager object at 0x7efc476bbd50>, type = None, value = None, traceback = None

    def __exit__(self, type, value, traceback):
        if type is None:
            try:
>               next(self.gen)
E               OSError: [Errno 29] Illegal seek

/home/stas/anaconda3/envs/main/lib/python3.7/contextlib.py:119: OSError
==================================================================== short test summary info =====================================================================
ERROR tests/test_trainer.py::TrainerIntegrationTest::test_trainer_eval_mrpc - OSError: [Errno 29] Illegal seek
FAILED tests/test_trainer.py::TrainerIntegrationTest::test_trainer_eval_mrpc - OSError: [Errno 29] Illegal seek
```

This is a possible solution, based on this solution
https://github.com/wandb/client/issues/1138#issuecomment-654943065


Trainer support for iterabledataset

To fix --> #5829


OpenAI GPT NoPaddingTokenFastTokenizerMatchingTest test fails

# 🐛 Bug

## Information

`tests/test_tokenization_fast.py::NoPaddingTokenFastTokenizerMatchingTest::test_all_tokenizers`
fails

## To reproduce

```
pytest   -n 1 tests/test_tokenization_fast.py::NoPaddingTokenFastTokenizerMatchingTest::test_all_tokenizers
================================================================ test session starts =================================================================
platform linux -- Python 3.7.5, pytest-5.4.3, py-1.9.0, pluggy-0.13.1
rootdir: /mnt/nvme1/code/huggingface/transformers-master
plugins: hypothesis-5.5.4, filter-subpackage-0.1.1, arraydiff-0.3, flaky-3.6.1, ipynb-1.1.1.dev0, cov-2.10.0, astropy-header-0.1.2, forked-1.2.0, doctestplus-0.5.0, openfiles-0.4.0, remotedata-0.3.2, xdist-1.32.0
gw0 [1]
F                                                                                [100%]
====================================================================== FAILURES ======================================================================
____________________________________________ NoPaddingTokenFastTokenizerMatchingTest.test_all_tokenizers _____________________________________________
[gw0] linux -- Python 3.7.5 /home/stas/anaconda3/envs/main/bin/python

self = <tests.test_tokenization_fast.NoPaddingTokenFastTokenizerMatchingTest testMethod=test_all_tokenizers>

    def test_all_tokenizers(self):
        for tok_case in self.TOKENIZERS_CLASSES:
            for pretrained_name in tok_case.python_cls.pretrained_vocab_files_map[tok_case.vocab_key].keys():

                # Tokenizer.filter makes it possible to filter which Tokenizer to case based on all the
                # information available in Tokenizer (name, rust class, python class, vocab key name)
                if tok_case.filter is None or (
                    tok_case.filter is not None and tok_case.filter(tok_case, pretrained_name)
                ):
                    kwargs = dict(t for t in tok_case.kwargs) if tok_case.kwargs else {}
                    with self.subTest("{} ({})".format(tok_case.name, pretrained_name)):
                        tokenizer_r = tok_case.rust_cls.from_pretrained(pretrained_name, **kwargs)
                        tokenizer_p = tok_case.python_cls.from_pretrained(pretrained_name, **kwargs)

>                       self.fast_align_python(tokenizer_r, tokenizer_p, tok_case, pretrained_name)

tests/test_tokenization_fast.py:62:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
tests/test_tokenization_fast.py:836: in fast_align_python
    self.assert_tokenization_python_rust_equals(tokenizer_r, tokenizer_p)
tests/test_tokenization_fast.py:192: in assert_tokenization_python_rust_equals
    self.assertSequenceEqual(input_p[key], input_r[key])
E   AssertionError: Sequences differ: [616,[111 chars] 0, 40477, 4830, 994, 580, 566, 260, 5958, 260[5290 chars] 239] != [616,[111 chars] 0, 4830, 994, 580, 566, 260, 5958, 260, 1490,[5160 chars] 239]
E
E   First differing element 29:
E   40477
E   4830
E
E   First sequence contains 12 additional elements.
E   First extra element 973:
E   1832
E
E   Diff is 8701 characters long. Set self.maxDiff to None to see it.
----------------------------------------------------------------- Captured log call ------------------------------------------------------------------
WARNING  transformers.tokenization_utils_base:tokenization_utils_base.py:2086 Token indices sequence length is longer than the specified maximum sequence length for this model (985 > 512). Running this sequence through the model will result in indexing errors
================================================================== warnings summary ==================================================================
/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/tensorflow_core/python/pywrap_tensorflow_internal.py:15
  /home/stas/anaconda3/envs/main/lib/python3.7/site-packages/tensorflow_core/python/pywrap_tensorflow_internal.py:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/directives.py:55
  /home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/directives.py:55: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    assert isinstance(locations, collections.Iterable), 'Must provide locations for directive.'

/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/directives.py:62
  /home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/directives.py:62: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    assert isinstance(args, collections.Mapping), '{} args must be a dict with argument names as keys.'.format(name)

/home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/typemap.py:1
  /home/stas/anaconda3/envs/main/lib/python3.7/site-packages/graphql/type/typemap.py:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
    from collections import OrderedDict, Sequence, defaultdict

-- Docs: https://docs.pytest.org/en/latest/warnings.html
============================================================== short test summary info ===============================================================
FAILED tests/test_tokenization_fast.py::NoPaddingTokenFastTokenizerMatchingTest::test_all_tokenizers - AssertionError: Sequences differ: [616,[111 ...
=========================================================== 1 failed, 4 warnings in 5.44s ============================================================
```

## Expected behavior

not fail ;)

I started digging and it fails only for `OpenAI GPT` tokenizer and passes `GPT2` tests.

If someone could give me a pointer, I'd be happy to investigate more.

I'm not sure whether it's the python or rust implementation that is at fault.

I tested that if I force max_length=512, to remove the warning, the test still fails in the same way.

## Environment info

```
- `transformers` version: 3.0.2
- tokenizers: 0.8.1rc1
- Platform: Linux-4.15.0-109-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.5
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): 2.0.1 (False)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```


[wip] T5 tokenizer should add special tokens


minor doc fixes

correct superclass name and small grammar fixes

then later added a correction in the error message:

It appears to be `BaseTokenizer` from looking at:

`from tokenizers.implementations import BaseTokenizer as BaseTokenizerFast`

and not `Tokenizer` as it currently says.

p.s. Please let me know whether you prefer specific separate PRs or bundled PRs for related things.


[cleanups] make Marian save as Marian

allow `MarianTokenizer.prepare_translation_batch` to ignore kwargs like `src_lang`


# 🐛 Bug

## Information

This is in regards to the `Trainer` object. The `get_train_dataloader` function uses RandomSampler or Distributed sampler, both of which I believe won't work with an [iterable datasets](https://pytorch.org/docs/stable/data.html#iterable-style-datasets) - which are only compatible with InfiniteConstantSampler see [pytorch documentation](https://pytorch.org/docs/stable/_modules/torch/utils/data/dataloader.html#DataLoader)

https://github.com/huggingface/transformers/blob/d088d744adb4e5aa45262a34acab3ae9e81de169/src/transformers/trainer.py#L229-L242

## Steps to reproduce the behavior:
1. Create an Iterable dataset and pass it to trainer object.

Gives the following error:
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<timed eval> in <module>

~/.conda/envs/my_root/lib/python3.6/site-packages/transformers/trainer.py in train(self, model_path)
    382                 training will resume from the optimizer/scheduler states loaded here.
    383         """
--> 384         train_dataloader = self.get_train_dataloader()
    385         if self.args.max_steps > 0:
    386             t_total = self.args.max_steps

~/.conda/envs/my_root/lib/python3.6/site-packages/transformers/trainer.py in get_train_dataloader(self)
    247             sampler=train_sampler,
    248             collate_fn=self.data_collator,
--> 249             drop_last=self.args.dataloader_drop_last,
    250         )
    251

~/.conda/envs/my_root/lib/python3.6/site-packages/torch/utils/data/dataloader.py in __init__(self, dataset, batch_size, shuffle, sampler, batch_sampler, num_workers, collate_fn, pin_memory, drop_last, timeout, worker_init_fn, multiprocessing_context)
    177                 raise ValueError(
    178                     "DataLoader with IterableDataset: expected unspecified "
--> 179                     "sampler option, but got sampler={}".format(sampler))
    180             elif batch_sampler is not None:
    181                 # See NOTE [ Custom Samplers and IterableDataset ]

ValueError: DataLoader with IterableDataset: expected unspecified sampler option, but got sampler=<torch.utils.data.sampler.RandomSampler object at 0x7fff5803b2b0>

## Expected behavior

Ideally, we should check if the dataset is an iterable dataset and if so, pass no sampler to the dataloader. Or allow the option of passing in our own sampler to the trainer.

## Environment info

- `transformers` version: 3.0.2
- Platform: Linux-4.4.121-92.104-default-x86_64-with-SuSE-12-x86_64
- Python version: 3.6.10
- PyTorch version (GPU?): 1.5.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: NO
- Using distributed or parallel set-up in script?: NO

I can contribute and create a PR to fix this. Let me know.


language tag addition on albert-mongolian

Related to pretrained model cards

hello!
I just added the language tag to model card. Can you guys check for me?


Reproducibility when using pretrained GPT2

# 🚀 Feature request

<!-- A clear and concise description of the feature proposal.
     Please provide a link to the paper and code in case they exist. -->

Ensure reproducibility when loading pretrained models.

## Motivation

<!-- Please outline the motivation for the proposal. Is your feature request
     related to a problem? e.g., I'm always frustrated when [...]. If this is related
     to another GitHub issue, please link here too. -->

In order to distribute results in a transparent manner, it is important to ensure reproducibility.

When loading a pretrained model, I cannot get the same weights, probably due to non-initialized weights.
Even when using `set_seed`, I cannot get twice the same models.

## Your contribution

<!-- Is there any way that you could help, e.g. by submitting a PR?
     Make sure to read the CONTRIBUTING.MD readme:
     https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md -->

I created a [small example](https://colab.research.google.com/gist/borisdayma/efa9ce5e8c7078bf12031b525f21f107/transformers-repeatability.ipynb) to illustrate the issue.


Vocab size mismatch on EncoderDecoder model from_pretrained

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): EncoderDecoder with bert-base-multilingual-cased in both

Language I am using the model on (English, Chinese ...): not relevant for the bug

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is: Not relevant for the bug

## To reproduce

Steps to reproduce the behavior:

I am trying to load a training checkpoint using the `save_pretrained` and `from_pretrained` API with the EncoderDecoder model. `EncoderDecoderModel.from_pretrained` fails to load the model when the configuration is loaded from the previously checkpointed model. I believe it's because it is loading a default vocab size (30522) instead of whatever is defined in the saved config (119547) in my case. To repro this run:

```
from transformers import EncoderDecoderModel, BertTokenizer, BertConfig, EncoderDecoderConfig

# Loading encoder-decoder model and saving it
load_dir = 'bert-base-multilingual-cased'
encoder_config = BertConfig.from_pretrained(load_dir)
decoder_config = BertConfig.from_pretrained(load_dir, is_decoder=True)
print(encoder_config.vocab_size)
print(encoder_config.vocab_size)
tokenizer = BertTokenizer.from_pretrained(load_dir)
model = EncoderDecoderModel.from_encoder_decoder_pretrained(load_dir, load_dir, encoder_config=encoder_config, decoder_config=decoder_config) # initialize Bert2Bert
model.save_pretrained('ok')

# Loading saved model and its configuration
encoder_config = BertConfig.from_pretrained('ok')
decoder_config = BertConfig.from_pretrained('ok')
print(encoder_config.vocab_size)
print(encoder_config.vocab_size)
encoder_decoder_config = EncoderDecoderConfig.from_encoder_decoder_configs(encoder_config, decoder_config)
model2 = EncoderDecoderModel.from_pretrained('ok', config=encoder_decoder_config) # This throws
```

The exception is the following:

```
File "/home/ancruzsa/.local/lib/python3.6/site-packages/transformers/modeling_utils.py", line 781, in from_pretrained
    model.__class__.__name__, "\n\t".join(error_msgs)
RuntimeError: Error(s) in loading state_dict for EncoderDecoderModel:
        size mismatch for encoder.embeddings.word_embeddings.weight: copying a param with shape torch.Size([119547, 768]) from checkpoint, the shape in current model is torch.Size([30522, 768]).
        size mismatch for decoder.bert.embeddings.word_embeddings.weight: copying a param with shape torch.Size([119547, 768]) from checkpoint, the shape in current model is torch.Size([30522, 768]).
        size mismatch for decoder.cls.predictions.bias: copying a param with shape torch.Size([119547]) from checkpoint, the shape in current model is torch.Size([30522]).
        size mismatch for decoder.cls.predictions.decoder.weight: copying a param with shape torch.Size([119547, 768]) from checkpoint, the shape in current model is torch.Size([30522, 768]).
        size mismatch for decoder.cls.predictions.decoder.bias: copying a param with shape torch.Size([119547]) from checkpoint, the shape in current model is torch.Size([30522]).
```

## Expected behavior

`from_pretrained(path)` should load the model without issues and using the provided configuration.

Edit: I was expecting `from_pretrained` with a single path as argument to work as explained in [#4595 comment](https://github.com/huggingface/transformers/issues/4595#issuecomment-638077144). However, it seems like doing `EncoderDecoderModel.from_encoder_decoder_pretrained('ok', 'ok', encoder_config=encoder_config, decoder_config=decoder_config)` does not throw an exception but it gives different results in text generation compared to `EncoderDecoderModel.from_pretrained(path)`. It would be great to confirm if both are supported and load the model weights correctly.

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2
- Platform: Linux
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.0 / Yes with GPU
- Tensorflow version (GPU?): None
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Add inference widget examples


New Community NB Add

Signed-off-by: lordtt13 <thakurtanmay72@yahoo.com>


Add model card for dv-wave

Related to pretrained model cards

Dhivehi / Maldives language model - includes links to notebooks for training and comparing performance on a given news classification task


[seq2seq] test that finetune takes < 7 seconds

Could also check memory and loss.


Create README.md

Related to pretrained model cards


Migrate Marian models names to ISO-639-3 where possible

Proposed renamings:

ROMANCE-> roa
CELTIC -> cel
NORWAY -> no  (check for conflicts w existing bilingual norwegian models.)
remove ch_group

@julien-c for NORTH_EU, why is the code ["multilingual"](https://github.com/huggingface/moon-landing/blob/d50efc4725a7546fb8159626470f3babacd44911/server/lib/ModelInfo.ts#L134-L170)
Should it be the same for all groups that do not map cleanly to an iso-639-3 group?

I will do this via mv, not cp, unless anyone objects.
cc @julien-c

Unresloved:
how will people know if a model requires a target language code?
we will still have many cases where model name is not a language code.


[seq2seq] pack_dataset.py rewrites dataset in max_tokens format

Inspired by fairseq max_tokens_dataset.
This works on disk, by regenerating the data directory, so it can work with or without @Pradhy729 's PR.
It is especially useful for avoiding padding computation in a multigpu setup, where SortishSampler doesn't work.

It will also play nice with TPU by allowing us to always of the same `input_ids` shape, rather than trimming batches to avoid padding computation.

Empirically, it reduces epoch time for mbart finetuning on 8GPU from 2.5hrs to 1.5 hrs. (You can increase batch size to 8 from 4), by cutting the number of examples by a factor of 2, and spending a bit more time on each example.

It also doesn't lead to that much more truncation than the previous approach, although I haven't quantified this.

### Example
```
wget https://s3.amazonaws.com/datasets.huggingface.co/translation/wmt_en_ro_128.tgz
tar -czvf wmt_en_ro_128.tgz
```
Creates a directory called `wmt_en_ro_packed/` with fewer examples

### Usage
```bash
python pack_dataset.py --tok_name facebook/mbart-large-en-ro --max_seq_len 128 --data_dir wmt_en_ro --save_path wmt_en_ro_packed_128
```
Output:
```bash
100%|██████████████████████████████████████████| 1999/1999 [00:01<00:00, 1533.19it/s]
packed val split from 1999 examples -> 683.
100%|██████████████████████████████████████████| 1999/1999 [00:01<00:00, 1590.80it/s]
packed test split from 1999 examples -> 652.
```


[seq2seq] Don't copy self.source in sortishsampler


error with transformers 2.9.1 but not with 2.3.0, same code, why?

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): Bert

Language I am using the model on (English, Chinese ...): Chinese

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

My problem is `return encoded` Error with transformers-2.9.1! but with transformers-2.3.0 was fine! Which way is Right? Very Confused!

My code as following:
```
# -*- coding: utf-8 -*-

import torch
from transformers import BertTokenizer, BertModel
from torch.utils.data import Dataset, DataLoader
from functools import partial
import logging

logging.basicConfig(level=logging.INFO)

bert_path = "/Users/kiwi/Desktop/chinese_wwm_ext"
model = BertModel.from_pretrained(bert_path)
tokenizer = BertTokenizer.from_pretrained(bert_path)

def tok_collate(batch_data):
    batch_sentence = [x[0] for x in batch_data]
    encoded = tokenizer.batch_encode_plus(
        batch_sentence,
        add_special_tokens=True,
        return_tensors='pt',
        pad_to_max_length=True)

    #return encoded['input_ids'], encoded['token_type_ids'], encoded['attention_mask']

    # (tensor([[ 101,  704, 1066,  704, 1925, 2600,  741, 6381,  510, 1744, 2157,  712,
    #          2375,  510,  704, 1925, 1092, 1999,  712, 2375,  739, 6818, 2398, 8108,
    #          3189,  683, 7305, 6626, 3959, 1266, 4689, 3636, 3727, 2356, 5440, 2175,
    #          4554, 2658, 7344, 2971, 2339,  868,  102]]), tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
    #          0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]), tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
    #          1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]]))

    return encoded
    #
    #   File "/Users/kiwi/anaconda/python.app/Contents/lib/python3.6/site-packages/transformers/tokenization_utils.py", line 203, in __getattr__
    #     return self.data[item]
    #   KeyError: '__getstate__'

def data_loader(data):
    dl = DataLoader(data, batch_size=8, shuffle=False, collate_fn=partial(tok_collate),
                    num_workers=2)

    for batch_data in dl:
        print(batch_data)

data = [('中共中央总书记、国家主席、中央军委主席习近平10日专门赴湖北省武汉市考察疫情防控工作', 1)]
data_loader(data)

```

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
```
encoded = {'input_ids': tensor([[ 101,  704, 1066,  704, 1925, 2600,  741, 6381,  510, 1744, 2157,  712,
         2375,  510,  704, 1925, 1092, 1999,  712, 2375,  739, 6818, 2398, 8108,
         3189,  683, 7305, 6626, 3959, 1266, 4689, 3636, 3727, 2356, 5440, 2175,
         4554, 2658, 7344, 2971, 2339,  868,  102]]), 'token_type_ids': tensor([[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
         1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}
```
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:2.9.1
- Platform:
- Python version:3.6+
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:


Additional layers to BERT

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
I'm currently fine-tuning BERTForSequenceClassification model for a classification task and I wanted to know if there are ways to add additional layers before the final classification layer?
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->


cast_bool_to_primitive breaks TensorFlow graph support.

# 🐛 Bug

## To reproduce

```python
import transformers
bert = tf.function(transformers.TFBertForMaskedLM.from_pretrained('bert-base-uncased'))

for i in range(2):
    (_, hidden_state) = bert(tf.constant([[10,11,12]]), output_hidden_states=True)
    print(f'computed {i}')
```

Errors with

```
ValueError: not enough values to unpack (expected 2, got 1)
```

## Expected behavior

```
computed 1
computed 2
```

Same result as if `tf.function` was not used.

## Environment info

Example environment : https://colab.research.google.com/gist/AndreasMadsen/593df94a3319dee58bba33a26efedeb3/untitled6.ipynb

- `transformers` version: 3.0.2
- Platform: Linux-4.19.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1+cu101 (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

-----

## Details

The bug happens due to `cast_bool_to_primitive`, that was introduced in https://github.com/huggingface/transformers/commit/6e603cb7892b49a2cbbc10ba859759f92c3fb7a6. Before that, it was possible to get the `hidden_states` from Bert in TensorFlow graph/function mode.

Generally speaking, casting TensorFlow tensors to primitives is not a good practice, as it only works in eager mode. It is also completely unnecessary in this case, as using `if bool_tensor_scalar:` works perfectly fine.

 ```python
def print_bool(x):
    if x:
        print('True')
    else:
        print('False')

print_bool_graph = tf.function(print_bool)

print('eager:')
print_bool(True) # Prints True
print_bool(False) # Prints False
print_bool(tf.constant(True)) # Prints True
print_bool(tf.constant(False)) # Prints False

print('')
print('graph:')
print_bool_graph(True) # Prints True
print_bool_graph(False) # Prints False
print_bool_graph(tf.constant(True)) # Prints True
print_bool_graph(tf.constant(False)) # Prints False
```

I can see there are some cases where defaults are used. The right way to handle that is to implement the default handling upstream in the first `call()` method. A lesser way would be to implement it as:

```python
def cast_bool_to_primitive(x, default_value=False):
  if x is None:
    return default_value
  return x
```


How to download original weights of gpt2

# ❓ Questions & Help
Hi, I have been trying to reproduce losses from [this paper](https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf), but I realized that my losses are lower than the ones from the paper. Also, I noticed a warning - Some weights of GPT2LMHeadModel were not initialized from the model checkpoint at gpt2-medium and are newly initialized: ... Does it mean that the original weights of GPT-2 was changed? If so, how can I download the original weights?
For instance my PPL for PTB is 35.20 while the PPL in the paper is 65.85.


Update README.md

Related to pretrained model cards

Fix missig "-" in meta data


[Longformer] fix longformer slow-down

A drastic slow-down of Longformer after the PR: https://github.com/huggingface/transformers/pull/5219 was deteced by @HHousen (thanks a lot!) here: https://github.com/huggingface/transformers/issues/4406#issuecomment-658483050 .

After digging a bit into the code it can be seen that the line:
`is_global_attn = any(is_index_global_attn.flatten())` is the reason of the drastic slow-down.

Running the following benchmark on master:
```
python examples/benchmarking/run_benchmark.py --models allenai/longformer-base-4096 --no_memory --sequence_length 512 1024
```

yields the following result:

```
====================       INFERENCE - SPEED - RESULT       ====================
--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length     Time in s
--------------------------------------------------------------------------------
 allenai/longformer-base-4096        8              512            0.647
 allenai/longformer-base-4096        8              1024           1.284
--------------------------------------------------------------------------------

====================        ENVIRONMENT INFORMATION         ====================
- transformers_version: 3.0.2
- framework: PyTorch
- use_torchscript: False
- framework_version: 1.5.0
- python_version: 3.7.7
- system: Linux
- cpu: x86_64
- architecture: 64bit
- date: 2020-07-16
- time: 14:07:02.719531
- fp16: False
- use_multiprocessing: True
- only_pretrain_model: False
- cpu_ram_mb: 32089
- use_gpu: True
- num_gpus: 1
- gpu: TITAN RTX
- gpu_ram_mb: 24217
- gpu_power_watts: 280.0
- gpu_performance_state: 0
- use_tpu: False
```

Now on this branch the results are as follows:

```
====================       INFERENCE - SPEED - RESULT       ====================
--------------------------------------------------------------------------------
          Model Name             Batch Size     Seq Length     Time in s
--------------------------------------------------------------------------------
 allenai/longformer-base-4096        8              512            0.141
 allenai/longformer-base-4096        8              1024           0.271
--------------------------------------------------------------------------------

====================        ENVIRONMENT INFORMATION         ====================
- transformers_version: 3.0.2
- framework: PyTorch
- use_torchscript: False
- framework_version: 1.5.0
- python_version: 3.7.7
- system: Linux
- cpu: x86_64
- architecture: 64bit
- date: 2020-07-16
- time: 14:07:57.623378
- fp16: False
- use_multiprocessing: True
- only_pretrain_model: False
- cpu_ram_mb: 32089
- use_gpu: True
- num_gpus: 1
- gpu: TITAN RTX
- gpu_ram_mb: 24217
- gpu_power_watts: 280.0
- gpu_performance_state: 0
- use_tpu: False
```

So this simple line is responsible for a slow-down of factor 4.

Moral of the story: Never use `any()` on a PyTorch tensor => always use tensor.any(). I wonder if this is actually a known problem of PyTorch/Python. It might be a good to check our code if we have more statements like this.

Another lesson for me is that one should always run the benchmarking before and after doing such a big refactoring as in https://github.com/huggingface/transformers/pull/5219 .
It's very simple to run the benchmark script for a model and takes usually only a couple of seconds. Ideally we should have performance regression tests to automatically detect such slow downs.

Pinging @thomwolf @mfuntowicz @sshleifer @LysandreJik @ibeltagy - think this is quite interesting.


Add missing arguments for BertWordPieceTokenizer

Hi:)

It seems that in `__init__` of `BertTokenizerFast`, arguments of `pad_token` and `mask_token` of `BertWordPieceTokenizer` are missing.

I've added them in this commit. If these missed arguments are intended, please let me know:)

Thank you!

```python
class BertTokenizerFast(PreTrainedTokenizerFast):
    def __init__(
        self,
        vocab_file,
        do_lower_case=True,
        unk_token="[UNK]",
        sep_token="[SEP]",
        pad_token="[PAD]",
        cls_token="[CLS]",
        mask_token="[MASK]",
        clean_text=True,
        tokenize_chinese_chars=True,
        strip_accents=None,
        wordpieces_prefix="##",
        **kwargs
    ):
        super().__init__(
            BertWordPieceTokenizer(
                vocab_file=vocab_file,
                unk_token=unk_token,
                sep_token=sep_token,
                cls_token=cls_token,
                clean_text=clean_text,
                handle_chinese_chars=tokenize_chinese_chars,
                strip_accents=strip_accents,
                lowercase=do_lower_case,
                wordpieces_prefix=wordpieces_prefix,
            ),
            unk_token=unk_token,
            sep_token=sep_token,
            pad_token=pad_token,
            cls_token=cls_token,
            mask_token=mask_token,
            **kwargs,
        )
```


I'm getting the above error while training for EncoderDecoder Model for longformer.

Following is the code snippet from the training loop, followed by the model definition.

`    output = model(input_ids = b_input_ids, attention_mask = b_input_masks, decoder_input_ids = b_decoder_input_ids, decoder_attention_mask = b_decoder_input_masks )
`
`model = EncoderDecoderModel.from_encoder_decoder_pretrained('allenai/longformer-base-4096','allenai/longformer-base-4096')
`
The crazy thing is I've not even defined 'head_mask' and left it to take its default value of 'none'.

Following the complete error.
`---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-18-68f3d2859ad0> in <module>()
     32     optimizer.zero_grad()
     33
---> 34     output = model(input_ids = b_input_ids, attention_mask = b_input_masks, head_mask = None,decoder_input_ids = b_decoder_input_ids, decoder_attention_mask = b_decoder_input_masks )
     35     loss = output[0]
     36

2 frames
/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

TypeError: forward() got an unexpected keyword argument 'head_mask'`


[Benchmark] Fix models without `architectures` param in config

This PR fixes a bug, when running benchmarks for models that have `config.architectures = None`, *e.g.* `longformer-base-4096`.


While running finetune.py in seq2seq examples on a custom dataset, I am getting the following error.

### This is what I'm getting:
Can I get some help? Thank you

Epoch 1:  34% 3410/10000 [1:43:10<3:19:22,  1.82s/it, loss=3.052, v_num=6]Traceback (most recent call last):
  File "finetune.py", line 344, in <module>
    main(args)
  File "finetune.py", line 322, in main
    logger=logger,
  File "/content/drive/My Drive/Colab Notebooks/transformers/examples/lightning_base.py", line 339, in generic_train
    trainer.fit(model)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 918, in fit
    self.single_gpu_train(model)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/distrib_parts.py", line 176, in single_gpu_train
    self.run_pretrain_routine(model)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/trainer.py", line 1093, in run_pretrain_routine
    self.train()
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/training_loop.py", line 375, in train
    self.run_training_epoch()
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/training_loop.py", line 458, in run_training_epoch
    _outputs = self.run_training_batch(batch, batch_idx)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/training_loop.py", line 634, in run_training_batch
    loss, batch_output = optimizer_closure()
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/training_loop.py", line 598, in optimizer_closure
    output_dict = self.training_forward(split_batch, batch_idx, opt_idx, self.hiddens)
  File "/usr/local/lib/python3.6/dist-packages/pytorch_lightning/trainer/training_loop.py", line 773, in training_forward
    output = self.model.training_step(*args)
  File "finetune.py", line 131, in training_step
    loss_tensors = self._step(batch)
  File "finetune.py", line 126, in _step
    outputs = self(source_ids, attention_mask=source_mask, decoder_input_ids=y_ids, labels=lm_labels,)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "finetune.py", line 112, in forward
    return self.model(input_ids, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_t5.py", line 1175, in forward
    return_tuple=return_tuple,
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_t5.py", line 692, in forward
    input_ids = input_ids.view(-1, input_shape[-1])
RuntimeError: cannot reshape tensor of 0 elements into shape [-1, 0] because the unspecified dimension size -1 can be any value and is ambiguous
Epoch 1:  34%|███▍      | 3410/10000 [1:43:10<3:19:23,  1.82s/it, loss=3.052, ### v_num=6]


BERT-viewer is broken for russian?

# 🐛 Bug
Hi I'm looking at BERT demos for russian, for example:
https://huggingface.co/DeepPavlov/rubert-base-cased?text=%D0%AF+%5BMASK%5D+%D1%81%D0%B0%D0%BB%D0%B0%D1%82+%D0%BD%D0%B0+%D0%BE%D0%B1%D0%B5%D0%B4.

English version for comparison:
https://huggingface.co/bert-base-cased?text=I+%5BMASK%5D+salad+for+lunch.

In the first link all token scores are 0.0 and they are absolute rubbish. Is everything fine with the viewer or the model?


[Fix] Checkpoint saving and I/O for XLNetQASimple

**1.  [Fix] Create directories for model checkpoints**

Saving model checkpoints without making directories would lead to error.
(i.e.  The output directory will not exist using the currenct script.)

The [older version](https://github.com/huggingface/transformers/blob/3f5ccb183e3cfa755dea2dd2afd9abbf1a0f93b8/examples/run_squad.py#L193) of the part of this script works fine.

**2.  [Fix] Fix wrong input/output structures for XLNet**

Running XLNet using the current script would lead to this error below.
> TypeError: forward() got an unexpected keyword argument 'cls_index'

The reason is that the [AutoModelForQuestionAnswering](https://github.com/huggingface/transformers/blob/ba2400189b2242620868096ae49babf93bd9ce00/examples/question-answering/run_squad.py#L735) would initialize XLNetForQuestionAnsweringSimple, while the I/O structures are written for XLNetForQuestionAnswering.


Add MPNet


Hosted Inference API: Error loading tokenizer Can't load config

Hi,

[https://huggingface.co/sampathkethineedi/industry-classification-api](https://huggingface.co/sampathkethineedi/industry-classification-api)

I uploaded my classification model fine tuned on BERT. There is no issue running the model from the hub and using the ‘sentiment-analysis’ pipeline. But there seems to be some problem when it comes to Hosted Inference API.

```
Error loading tokenizer Can't load config for 'sampathkethineedi/industry-classification-api classification'. Make sure that: - 'sampathkethineedi/industry-classification-api' is a correct model identifier listed on 'https://huggingface.co/models' - or 'sampathkethineedi/industry-classification-api' is the correct path to a directory containing a config.json file OSError("Can't load config for 'sampathkethineedi/industry-classification-api'. Make sure that:\n\n- 'sampathkethineedi/industry-classification-api' is a correct model identifier listed on 'https://huggingface.co/models'\n\n- or 'sampathkethineedi/industry-classification-api' is the correct path to a directory containing a config.json file\n\n")
```

Can someone help me with this?

Thanks!


[WIP - Benchmark] Add generate function

This PR adds the `generate` function to Benchmarks. You can try it out by running:

```
python examples/benchmarking/run_benchmark.py --models gpt2 --batch_sizes 1 --sequence_lengths 20 --no_inference --generate
```

## TODO
- [ ] Add for TF as well


[Benchmark] fix benchmark non standard model

This PR fixes a typo in Benchmark, that allows to load specifc architectures and not just the base model.


GPT2 weights don't initialize from checkpoint

OS: OSX 10.15.5 (Catalina)
Transformers version 3.0.2

I'm running into this warning when I'm trying to initialize a pre-trained GPT2 model.
This looks a bit worrying to me because it looks like like it ignores all the pre-trained attention heads, or am I missing something here?

<img width="1087" alt="Screenshot 2020-07-16 at 11 19 10" src="https://user-images.githubusercontent.com/8694790/87654081-b837c900-c756-11ea-925e-106314ad9942.png">

Any idea of what's gone wrong? I've been running the same code on another computer earlier without encountering this problem, but on my current setup I haven't been able to get around it. I also tried deleting the downloaded files from the cache and re-loading the models with no luck.

Any help is very appreciated!


Issue when load pretrained weights

I got the following error when running

AutoModelWithLMHead.from_pretrained("bert-base-chinese")

OSError: Can't load weights for 'bert-base-chinese'. Make sure that:

- 'bert-base-chinese' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'bert-base-chinese' is the correct path to a directory containing a file named one of pytorch_model.bin, tf_model.h5, model.ckpt.


Lightning Updates for v0.8.5

Fixing #5361 ...battling with unittests.


# ❓ Can I use the pretrained BERT-Base model directly for predict isNextSentence task?

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
I has a document-level corpus, but it don't have document boundaries.
I want to confirm that whether can I predict isNextSentence task by pretrained BERT-Base model? The model does not use any data finetune and don't mask any token when i used it.
Is prediction reliable by this way?


Moving transformers package import statements to relative imports in some files

When using the transformers library as a local submodule (eg git submodule) instead of a python package, it's important to have relative import instead of doing `from transformers` directly which would look at the installed version of the python package. Regardless, it seems like the codebase favors relative imports in general, but it seems a few cases were not added that way.

This pull request moves to relative paths occurrences in some files under the `src` folder, except by comments and the `convert_*` files which are likely importing the python package intentionally.


LongFormerAttention For AutoRegressive Models

Longformer Currently supports only Bidirectional Attention,It would be a great feature to finetune current language models like GPT-2 on longer sequences .


Print all next tokens of a sentence over a certain probability threshold.

How would I do this using GPT-2?


Adding the LXMERT pretraining model (MultiModal  languageXvision)  to HuggingFace's suite of models

LXMERT is considered a dual-stream language-vision model. Dual stream meaning that it uses a transformer encoder to perform self-attention between each modality of data and then a cross-modality transformer encoder for a fine-grained cross-attention between each modality. It's achieved tremendous (STOTA results) success across a wide variety of downstream tasks (GQA, VQA2, NLVR2).

Here is the original link to the paper:  https://arxiv.org/pdf/1908.07490.pdf
Here is the link to the original implementation: https://github.com/airsplay/lxmert
and here is the weight to the original model weights: https://nlp1.cs.unc.edu/data/model_LXRT.pth

Please let me know if there is anything i missed, and I would be very grateful for any help if I end up running into any blockers, but I will do my best to follow the detailed instructions in the templates.

This is also a work in progress!


Seq2SeqDataset uses linecache to save memory by @Pradhy729 (#5792)


Add script to convert tf2.x checkpoint to PyTorch

The script converts the newer TF2.x checkpoints (as published on their [official GitHub](https://github.com/tensorflow/models/tree/master/official/nlp/bert) to Pytorch. The [existing script](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_bert_original_tf_checkpoint_to_pytorch.py) only works with checkpoints from the [original BERT repository](https://github.com/google-research/bert) which uses TF 1.4.

The script currently only converts the encoder part (but no MLM/NSP heads). The official checkpoints published by the tensorflow team unfortunately also don't contain these heads. I have written a script which takes care of these, but it does add a fair bit of complexity.

I have tested on my side by comparing all model weights with the official Huggingface version:

```python
from transformers import BertModel
import torch

def validate_model(bert_original, bert_converted):
    assert bert_original.num_parameters() == bert_converted.num_parameters()
    assert len(bert_original.state_dict()) == len(bert_converted.state_dict())
    for (layer_original, value_original), (layer_converted, value_converted) in zip(bert_original.state_dict().items(), bert_converted.state_dict().items()):
        assert layer_original == layer_converted
        if not torch.eq(value_original, value_converted).all():
            raise ValueError(f'Incorrect weights for {layer_original}')
    print('Success! Both models are identical!')

if __name__ == "__main__":
    validate_against = 'bert-base-uncased'
    path_to_converted_model = './converted_bert_base_uncased'
    bert_converted = BertModel.from_pretrained(path_to_converted_model)
    bert_original = BertModel.from_pretrained(validate_against)
    validate_model(bert_original, bert_converted)
```

I'm happy to write some tests for this if needed (and if possible) or have any other input.


github issue template suggests who to tag

Fewer issues get lost if people tag the relevant developer. It also saves @LysandreJik time.
While he is out, I figured we could experiment with trying to nudge issue raisers to tag.

Here is the comment at the beginning of the "Bug Report" template:

Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of who to tag.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @julien-c
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 blenderbot: @mariamabarham
 Bart: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 tensorflow: @jplu

I wrote it in 3 minutes in case people hate this idea, so I am sure it is missing people!
Suggestions very much appreciated.


Update README.md

Related to pretrained model cards

Created PyTorch version of model. Minor update on README.


add attention_dropout, relu_dropout command line args to lightning_base.py

then pass them to config in `__init__`.


Can't load weights for GPT2 error

## System Info
Pop!_OS 20.04
Pytorch: 1.5.1
Transformers: 2.11.0
Python: 3.7.6

## Details
I am working behind a proxy. If I run the following:
```python
from transformers import GPT2Tokenizer
proxies = {'http':'http://my.proxy.com:port', 'https':'https://my.proxy.com:port'}
tokenizer = GPT2Tokenizer.from_pretrained("gpt2", proxies=proxies)
```
The tokenizer gets downloaded. However, if I run:
```python
from transformers import GPT2LMHeadModel
proxies = {'http':'http://my.proxy.com:port', 'https':'https://my.proxy.com:port'}
model = GPT2LMHeadModel.from_pretrained("gpt2", proxies=proxies)
```
I get the following error:
```python
Traceback (most recent call last):
  File "/path/to/anaconda3/lib/python3.7/site-packages/transformers/modeling_utils.py", line 638, in from_pretrained
    raise EnvironmentError
OSError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/path/to/text_gen_w_transformers/finetune_test.py", line 28, in <module>
    model = GPT2LMHeadModel.from_pretrained("gpt2", proxies=proxies)
  File "/path/to/anaconda3/lib/python3.7/site-packages/transformers/modeling_utils.py", line 645, in from_pretrained
    raise EnvironmentError(msg)
OSError: Can't load weights for 'gpt2'. Make sure that:

- 'gpt2' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'gpt2' is the correct path to a directory containing a file named one of pytorch_model.bin, tf_model.h5, model.ckpt.
```

Any thoughts about what might be the issue? Thanks in advance for your help!


Faster mBART finetuning

Examples

Which is related to examples in general

Help wanted

Extra attention is needed, help appreciated

Goal: Get BLEU 20 in 1 epoch on wmt-en-ro.
Can't run a bs=1 without `--freeze_embeds`.
1 epoch takes 6H on 16GB GPU with fp16, with `--freeze_embeds` and `--freeze_encoder`. Max bs=4

Ideas:

- [ ] Dataset that fits as many sentences as possible into an example, to increase gpu utilization.
- [ ] Only store embeddings once
- [ ] prune embeddings: https://github.com/pytorch/fairseq/issues/2120
- [ ] `label_smoothing=0.1`
- [ ] TPU?

Fairseq finetune command:
https://github.com/pytorch/fairseq/issues/2179


Sentence-transformers model outputs different than when loaded in HuggingFace

# 🐛 Bug

## Information

The outputs I'm seeing from generating sentence embeddings with the steps in the model zoo for getting sentence-transformers embeddings deviate from those generated from the SentenceTransformers package.

I'm assuming this is due to the lack of pooling. Is there a way to convert a ST model to HF with pooling?

English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Load a pre-trained model from SentenceTransformers
2. Generate sentence embeddings from tokenized inputs
3. Print sentence embeddings

HuggingFace:
```python
#Sentences we want sentence embeddings for
sentences = ['This framework generates embeddings for each input sentence',
             'Sentences are passed as a list of string.',
             'The quick brown fox jumps over the lazy dog.']
tokenizer = AutoTokenizer.from_pretrained("roberta-large-nli-stsb-mean-tokens/0_RoBERTa/")
>>> model = AutoModel.from_pretrained("roberta-large-nli-stsb-mean-tokens/0_RoBERTa/")
>>> encoded_input = tokenizer(sentences, padding=True, truncation=True, max_length=128, return_tensors='pt')
>>> with torch.no_grad():
...     model_output = model(**encoded_input)
...     sentence_embeddings = model_output[0][:,0]
...
>>> print("Sentence embeddings:")
Sentence embeddings:
>>> print(sentence_embeddings)
tensor([[ 0.0057, -0.7690,  0.0702,  ...,  0.0734, -1.4343,  0.3418],
        [ 0.2066, -0.8213,  0.1272,  ...,  0.2649, -1.2799, -0.1636],
        [-0.4860, -0.5176, -0.5924,  ..., -0.4880, -0.1880, -0.0554]])
```

SentenceTransformers

```python
model = SentenceTransformer('roberta-large-nli-stsb-mean-tokens')
sentence_embeddings = model.encode(sentences)>>> sentence_embeddings = model.encode(sentences)
>>> sentence_embeddings = model.encode(sentences)
>>> print(sentence_embeddings)
[array([ 0.6306487 , -0.2879937 ,  0.05334993, ...,  0.26865923,
       -2.2382815 ,  0.22505784], dtype=float32), array([ 0.22068763, -0.8045991 ,  0.18439776, ...,  0.6993382 ,
       -1.7670776 ,  0.11258417], dtype=float32), array([-0.17819108,  0.08762542, -0.7614953 , ..., -0.6983883 ,
       -0.13175072, -0.11123852], dtype=float32)]
```

## Expected behavior

SentenceTransformer and HF outputs should be the same

## Environment info

```
- `transformers` version: 3.0.2
- Platform: Darwin-18.7.0-x86_64-i386-64bit
- Python version: 3.7.5
- PyTorch version (GPU?): 1.5.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```


[fix] Style. Trying again


Marian Conversion Script

I want to utilize and train the machine translation models posted on Github: https://github.com/Helsinki-NLP/OPUS-MT-train/tree/master/models on my own corpus. These models are C++ based.

Are these models exactly the same as the ones posted by Hugging Face’s website: https://huggingface.co/Helsinki-NLP ?

If they are, what is the transformers conversion script that loads these Github models and transforms them to models that are loadable via transformers (for example utilizing: transformers.AutoTokenizer.from_pretrained(‘path’), transformers.AutoModelWithLMHead.from_pretrained(‘path’))?

@sshleifer @jackalhan


Create README.md

model card

Related to pretrained model cards


Error in conversion to tensorflow

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): DistilBERT

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

```
from transformers import TFAutoModel, AutoTokenizer, AutoModel
import os

model = AutoModel.from_pretrained('distilbert-base-uncased')
os.system('mkdir distilbert')
model.save_pretrained('distilbert')
model = TFAutoModel.from_pretrained('distilbert', from_pt=True) # crashes
```

## Expected behavior

Model is converted from pytorch to tensorflow

## Environment info
- `transformers` version: 3.0.2
- Platform: Linux-5.3.0-62-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1 (True)
- Tensorflow version (GPU?): 2.0.0-beta1 (False)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

## Actual Behaviour
```
Traceback (most recent call last):
  File "pt2tf.py", line 8, in <module>
    model = TFAutoModel.from_pretrained('distilbert', from_pt=True)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_auto.py", line 423, in from_pretrained
    return model_class.from_pretrained(pretrained_model_name_or_path, *model_args, config=config, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_utils.py", line 482, in from_pretrained
    return load_pytorch_checkpoint_in_tf2_model(model, resolved_archive_file, allow_missing_keys=True)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_pytorch_utils.py", line 93, in load_pytorch_checkpoint_in_tf2_model
    tf_model, pt_state_dict, tf_inputs=tf_inputs, allow_missing_keys=allow_missing_keys
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_pytorch_utils.py", line 125, in load_pytorch_weights_in_tf2_model
    tf_model(tf_inputs, training=False)  # Make sure model is built
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 712, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_distilbert.py", line 603, in call
    outputs = self.distilbert(inputs, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 712, in __call__
    outputs = self.call(inputs, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_distilbert.py", line 493, in call
    embedding_output = self.embeddings(input_ids, inputs_embeds=inputs_embeds)  # (bs, seq_length, dim)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 709, in __call__
    self._maybe_build(inputs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 1966, in _maybe_build
    self.build(input_shapes)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_tf_distilbert.py", line 112, in build
    "weight", shape=[self.vocab_size, self.dim], initializer=get_initializer(self.initializer_range)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer.py", line 389, in add_weight
    aggregation=aggregation)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/training/tracking/base.py", line 713, in _add_variable_with_custom_getter
    **kwargs_for_getter)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer_utils.py", line 154, in make_variable
    shape=variable_shape if variable_shape else None)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variables.py", line 260, in __call__
    return cls._variable_v1_call(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variables.py", line 221, in _variable_v1_call
    shape=shape)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variables.py", line 199, in <lambda>
    previous_getter = lambda **kwargs: default_variable_creator(None, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variable_scope.py", line 2502, in default_variable_creator
    shape=shape)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/variables.py", line 264, in __call__
    return super(VariableMetaclass, cls).__call__(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 464, in __init__
    shape=shape)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/resource_variable_ops.py", line 608, in _init_from_args
    initial_value() if init_from_fn else initial_value,
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/engine/base_layer_utils.py", line 134, in <lambda>
    init_val = lambda: initializer(shape, dtype=dtype)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/init_ops_v2.py", line 341, in __call__
    dtype = _assert_float_dtype(dtype)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/init_ops_v2.py", line 769, in _assert_float_dtype
    raise ValueError("Expected floating point type, got %s." % dtype)
ValueError: Expected floating point type, got <dtype: 'int32'>.
```


[bart] decoder.last_hidden_state shape changes when passing labels

```
config = BartConfig(
            vocab_size=99,
            d_model=24,
            encoder_layers=2,
            decoder_layers=2,
            encoder_attention_heads=2,
            decoder_attention_heads=2,
            encoder_ffn_dim=32,
            decoder_ffn_dim=32,
            max_position_embeddings=48,
            add_final_layer_norm=True,
        )
        lm_model = BartForConditionalGeneration(config).to(torch_device)
        context = torch.Tensor([[71, 82, 18, 33, 46, 91, 2], [68, 34, 26, 58, 30, 2, 1]]).long().to(torch_device)
        summary = torch.Tensor([[82, 71, 82, 18, 2], [58, 68, 2, 1, 1]]).long().to(torch_device)
        loss, logits, enc_features = lm_model(input_ids=context, decoder_input_ids=summary, labels=summary)

        expected_shape = (*summary.shape, config.vocab_size)
        self.assertEqual(logits.shape, expected_shape)
        outputs2 = lm_model(input_ids=context, decoder_input_ids=summary)
        self.assertEqual(outputs2.logits.shape, expected_shape)
# Fails torch.Size([2, 1, 99]) != (2, 5, 99)
```
Is this expected @sgugger ?


Error using DataParallel with reformer model: There were no tensor arguments to this function

# 🐛 Bug

## Information

I'm having some issues using DataParallel with the reformer model with 4 GPUs. I am trying to feed the ReformerModel input embeddings, and output the last hidden state. I am using apex amp, however I get the same error when I don't use amp. I also get the same error when I use input IDs, rather than embeddings. And I've tested the same script using other HuggingFace models with no issues (Bert, and Roberta).

## To reproduce

Simple code:

```
import torch
from apex import amp
import transformers
from transformers import ReformerModel
from torch.utils.data import TensorDataset, DataLoader
import torch.nn as nn
print(transformers.__version__)
print(torch.__version__)

device = torch.device("cuda:0")
batch_size = 4

model_rf = ReformerModel.from_pretrained('google/reformer-crime-and-punishment')
model_rf.to(device)
opt_rf = torch.optim.AdamW(model_rf.parameters(), lr=0.0002)

model_rf, opt_rf = amp.initialize(model_rf, opt_rf)
model_rf = nn.DataParallel(model_rf)
embeds = torch.randn(80, 64, 256)

training_set = TensorDataset(embeds, embeds)
training_generator = DataLoader(training_set, batch_size=batch_size, shuffle=True)

for i, batch in enumerate(training_generator):
    embeds, _ = batch
    h_final = model_rf(inputs_embeds=embeds.to(device))
```
And the error:

```
Traceback (most recent call last):
  File "rf_4.py", line 35, in <module>
    h_final = model_rf(inputs_embeds=embeds)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py", line 155, in forward
    outputs = self.parallel_apply(replicas, inputs, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/data_parallel.py", line 165, in parallel_apply
    return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/parallel_apply.py", line 85, in parallel_apply
    output.reraise()
  File "/usr/local/lib/python3.6/dist-packages/torch/_utils.py", line 395, in reraise
    raise self.exc_type(msg)
RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_reformer.py", line 1621, in forward
    embedding_output = self.embeddings(input_ids=input_ids, position_ids=position_ids, inputs_embeds=inputs_embeds)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_reformer.py", line 234, in forward
    position_embeddings = self.position_embeddings(position_ids)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/transformers/modeling_reformer.py", line 170, in forward
    [weight[:, :required_pos_encodings_columns] for weight in broadcasted_weights], dim=-1
  File "/usr/local/lib/python3.6/dist-packages/apex/amp/wrap.py", line 81, in wrapper
    return orig_fn(seq, *args, **kwargs)
RuntimeError: There were no tensor arguments to this function (e.g., you passed an empty list of Tensors), but no fallback function is registered for schema aten::_cat.  This usually means that this function requires a non-empty list of Tensors.  Available functions are [CPUTensorId, CUDATensorId, QuantizedCPUTensorId, VariableTensorId]
```
## Expected behavior

This code kicks an error at the h_final line

## Environment info

- `transformers` version: 3.0.2
- Platform: Ubuntu 18.04
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1
- Tensorflow version (GPU?): no
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: Yes, 4 GPUs


Bug in MiniLM-L12-H384-uncased modelhub model files

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): microsoft/MiniLM-L12-H384-uncased

Language I am using the model on (English, Chinese ...): English

The problem arises when using:
* [x] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: SQuAD ver.2
* [ ] my own task or dataset: (give details below)

Problem: The vocab for microsoft/MiniLM-L12-H384-uncased is missing a token => wrong tokenization => bad performance for SQuAD finetuning
Potential fix: Upload the original vocab that was published in the original Microsoft Repository (https://github.com/microsoft/unilm/tree/master/minilm)

## To reproduce

Steps to reproduce the behavior:
1. While tokenizing the a sample english sentence with miniLM model downloaded from modelhub
2. Comparing for modelhub tokenizer vocab size vs. modelhub model vocab size
```
from transformers import AutoTokenizer, AutoModel
tokenizer_modelhub = AutoTokenizer.from_pretrained("microsoft/MiniLM-L12-H384-uncased")
model_mod
elhub = AutoModel.from_pretrained("microsoft/MiniLM-L12-H384-uncased")
assert tokenizer_modelhub.vocab_size == model_modelhub.embeddings.word_embeddings.num_embeddings, "tokenizer vocab_size {} doesn't match embedding vocab size {} ".format(tokenizer.vocab_size, model.embeddings.word_embeddings.num_embeddings)
```
Output
```
AssertionError: tokenizer vocab_size 30521 doesn't match embedding vocab size 30522
```
3. Download "original" MiniLM model from Microsoft's MiniLM GitHub Repo (https://1drv.ms/u/s!AjHn0yEmKG8qixAYyu2Fvq5ulnU7?e=DFApTA)
4. Comparing the modelhub MiniLM tokenizer and "original" MiniLM tokenizer token ids
```
import torch
input_ids_modelhub = torch.tensor([tokenizer_modelhub.encode("Let's see all hidden-states and attentions on this text")])
config_github = AutoConfig.from_pretrained("<github_minilm_model_directory>")
tokenizer_github = AutoTokenizer.from_pretrained(
...            "<github_minilm_model_directory>", config=config_github)
model_github_finetuned = AutoModelForQuestionAnswering.from_pretrained(
...            "<github_minilm_model_directory>", config=config_github)
assert tokenizer_github.vocab_size == model_github.embeddings.word_embeddings.num_embeddings, "tokenizer vocab_size {} doesn't match embedding vocab size {} ".format(tokenizer.vocab_size, model.embeddings.word_embeddings.num_embeddings)
input_ids_github = torch.tensor([tokenizer_github.encode("Let's see all hidden-states and attentions on this text")])
```
```
print(input_ids_github)
tensor([[ 101, 2292, 1005, 1055, 2156, 2035, 5023, 1011, 2163, 1998, 3086, 2015,
         2006, 2023, 3793,  102]])
```
```
print(input_ids_modelhub)
tensor([[ 100, 2291, 1004, 1054, 2155, 2034, 5022, 1010, 2162, 1997, 3085, 2014,
         2005, 2022, 3792,  101]])
```
5. Fine-tune modelhub MiniLM model for SQuAD ver 2

```
python examples/question-answering/run_squad.py --model_type bert \
--model_name_or_path microsoft/Multilingual-MiniLM-L12-H384 \
--output_dir finetuned_modelhub_minilm \
--data_dir data/squad20 \
--train_file train-v2.0.json \
--predict_file dev-v2.0.json \
--learning_rate 4e-5 \
--num_train_epochs 4 \
--max_seq_length 384 \
--doc_stride 128 \
--per_gpu_train_batch_size 12 \
--per_gpu_eval_batch_size 12 \
--gradient_accumulation_steps 4 \
--version_2_with_negative \
--do_lower_case \
--verbose_logging \
--do_train \
--do_eval \
--seed 42 \
--save_steps 5000 \
--overwrite_output_dir \
--overwrite_cache
```
Results:
```
{'exact': 59.681630590415224, 'f1': 63.78250778488946, 'total': 11873, 'HasAns_exact': 49.73009446693657, 'HasAns_f1': 57.94360913123985, 'HasAns_total': 5928, 'NoAns_exact': 69.60470984020185, 'NoAns_f1': 69.60470984020185, 'NoAns_total': 5945, 'best_exact': 59.690053061568264, 'best_exact_thresh': 0.0, 'best_f1': 63.79093025604285, 'best_f1_thresh': 0.0}
```
6. Fine-tune original MiniLM model for SQuAD ver 2

```
python examples/question-answering/run_squad.py --model_type bert \
--model_name_or_path <saved_githubModel_local_path> \
--output_dir finetuned_github_minilm \
--data_dir data/squad20 \
--train_file train-v2.0.json \
--predict_file dev-v2.0.json \
--learning_rate 4e-5 \
--num_train_epochs 4 \
--max_seq_length 384 \
--doc_stride 128 \
--per_gpu_train_batch_size 12 \
--per_gpu_eval_batch_size 12 \
--gradient_accumulation_steps 4 \
--version_2_with_negative \
--do_lower_case \
--verbose_logging \
--do_train \
--do_eval \
--seed 42 \
--save_steps 5000 \
--overwrite_output_dir \
--overwrite_cache
```
Results:
```
{'exact': 76.23178640613156, 'f1': 79.57013365427773, 'total': 11873, 'HasAns_exact': 78.50877192982456, 'HasAns_f1': 85.1950399590485, 'HasAns_total': 5928, 'NoAns_exact': 73.96131202691338, 'NoAns_f1': 73.96131202691338, 'NoAns_total': 5945, 'best_exact': 76.23178640613156, 'best_exact_thresh': 0.0, 'best_f1': 79.57013365427775, 'best_f1_thresh': 0.0}
 ```

## Expected behavior
1. Assertions should pass.
2. `input_ids_modelhub` and `input_ids_github` should produce same results
3. Reproduce the Downstream results on MiniLM modelhub files as mentioned in MiniLM model card

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2
- Platform: Ubuntu 18.04.4 LTS
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1 (Yes)
- Tensorflow version (GPU?): Not using
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No


Update README.md

Related to pretrained model cards

Add cherry picked example for the widget


[squad] make examples and dataset accessible from SquadDataset object

In order to do evaluation on the SQuAD dataset using `squad_evaluate`, the user needs access to both the examples loaded in the dataset and the `TensorDataset` that contains values like `unique_id` and the like that are used in constructing the list of `SquadResult` objects. This PR surfaces the examples and dataset to the user so that they can access it directly.

For example of why access to those is needed, see how evaluation is currently done in `examples/run_squad.py`. The `SquadDataset` object attempts to wrap up some of this functionality, but without access to examples and dataset the evaluation is not possible.


[fix] check_code_quality


Ensure OpenAI GPT position_ids is correctly initialized and registered at init.

This will make it compatible with TorchScript export and avoid hardcoded `position_ids` tensor's device in the generated graph.

Solve the following issue: #5664

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>


[fix] check code quality


Fine-tune BERT for regression problem

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
I was wondering in order to fine-tune BERT for regression problem, do I just need to put num_labels = 1 inside the BertForSequenceClassification function? Does there anything else need to modify? I'm a newbie to transformers and still confused with basic things. Thanks in advance.
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


XLNet `use_cache` refactor

As discussed with @joeddav and @sgugger, this PR lightly refactors the `use_cache` argument in XLNet.

- In line with #5438, in the model methods `use_cache` defaults to None, which redirects to the model config value if no value is passed.
- `use_cache` is independent of `mem_len`: if `use_cache` is True and `mem_len` is 0 or None (which is the case in the base model config), the model behaves like GPT-2 and returns `mems` to be used as `past` in generation.
- This changes functionality and will enable the default model to use caching; for instance, it should speed up the [inference widget](https://huggingface.co/xlnet-base-cased?text=My+name+is+Clara+and+I+am+) significantly (x3 speed-up on my CPU)


[fix] T5 ONNX test: model.to(torch_device)

Should fix #5724

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>


Confidence score prediction of pretrained models in extractive QA - similar to pipeline

How can we calculate the confidence of a single sentence predicted from extractive question answering using autotokenizer. we will get a score using the pipeline method for a sentence, but what we get from the extractive qa are answer_Start_scores and answer_end_scores. How can we get one single score ?


How to get parameters from a Query.

# ❓ Questions & Help

Hi maybe someone can help me with a problem, I need to get parameters from a query.
I already use question answering pipeline to identify a report and now I want to get some parameters to use as filter, example:

Query: How many electric cars sold in 2019?

Output: (Item: eletric cars / Year:2019)


Hello,I have this problem in running 'run_glue.py'!

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
hello ,I have this problem in running 'run_glue.py'.Anyone can help me ?Thanks!
from transformers import glue_compute_metrics
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'glue_compute_metrics' from 'transformers' (/home/wangbingchen/wangbingchen/anaconda3/lib/python3.7/site-packages/transformers/__init__.py)

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on Stack Overflow**:


When using "transformers.WarmUp" with tensorflow 2.0.0, warming up restart in each epoch!
That is because, in keras "Callbacks.on_batch_begin(self, batch, logs)", batch start from zero in each epoch when using "fit" method.


TF Longformer

This PR adds Longformer

In a first step, it is made sure that code is clean and that all tests pass. Todo:

### ToDo List:

- [x] same output for local attention only
- [x] same output for local + global attention only
- [x] same output for aggressive test
- [x] add all other tests
- [x] add longformer QA
- [x] refactor code and run benchmark
- [x] adds weights to all QA models and check performance via notebook

### ToDo after PR is merged:
- [ ] Add Longformer for SeqClass, MC, ...  ("good first issue")
- [ ] Speed up performance and make GPU XLA work -> use Benchmark tools and possible TF Profiler

### For Review

This PR adds `TFLongformer` and the two most important parent classes `TFLongformerForMaskedLM` and `TFLongformerForQuestionAnswering`.  Many tests are added to verify that TFLongformer gives identical results to PT Longformer and a colab notebook (see below) is attached to show performance on a real task.

Below you can find a Benchmark showing that TFLongformer is about 1.5x slower than PT on GPU. For now this is acceptable IMO, but in a future PR I want to take a deeper look at how TF code can be optimized and also solve a problem there is currently with TF XLA.

I spent a lot of time, trying to solve this issue: https://github.com/huggingface/transformers/issues/5815 for TFLongformer and didn't manage to find a good solution. The corresponding tests are in `SLOW` mode so they won't fail on this PR. Since we are currently thinking about a better solution than using `cast_bool_to_primitive` to solve the know TF graph boolean error, I think I will leave this small bug in TFLongformer for now (it's quite an edge IMO anymays).

Docs are added and checked, comments are added, performance on TriviaQA is verified in TF colab: https://colab.research.google.com/drive/1UmU3T1nPmJ2LgXQtPcaEVtXUnoBJ1onF?usp=sharing and TF weights were added to all longformer models here: https://huggingface.co/models?search=longformer.

Would be happy about a review @jplu @ibeltagy @LysandreJik @sshleifer @sgugger @julien-c @thomwolf


ADD ERNIE model

Related to pretrained model cards

ERNIE is a serial model released by Baidu including:
ERNIE1.0: [Ernie: Enhanced representation through knowledge integration](https://arxiv.org/abs/1904.09223)
ERNIE2.0: [ERNIE 2.0: A Continual Pre-training Framework for Language Understanding](https://arxiv.org/abs/1907.12412)
ERNIE-tiny: Use data distillation from ERNIE 2.0
These models have state-of-the-art performances on NLU tasks, especially on Chinese tasks.
So, ERNIE is a very important model in the transformer model family.

Actually, ERNIE model is the same structure as BERT including the above three ERNIE viersions, so we don't add a new model, and just need to convert the weight.
> Note that: ERNIE2.0 introduces a task embedding, but the official released version doesn't have this embedding weight, and  all released weights are the same with BERT.

I have successfully converted  ERNIE models into PyTorch version and do serial experiments to check the conversion is the same with the original paddlepaddle implement.
More detail: https://github.com/nghuyong/ERNIE-Pytorch

In this PR, I directly add ERNIE model to BERT model related files.
This PR is link to issue: [#issue5117](https://github.com/huggingface/transformers/issues/5117),[#issue928](https://github.com/huggingface/transformers/issues/928) and [#issue514](https://github.com/huggingface/transformers/issues/514)


Feature request of Sparselty Gated Mixture-of-Experts and PowerNorm

# 🚀 Feature request

- The newer variant of MoE for Transformer as in [GShard: Scaling Giant Models with Conditional
Computation and Automatic Sharding](https://arxiv.org/abs/2006.16668) ([Relevant codes](https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/models/research/moe.py))
- PowerNorm of [PowerNorm: Rethinking Batch Normalization in Transformers](https://arxiv.org/abs/2003.07845) ([Relevant codes](https://github.com/sIncerass/powernorm/blob/master/fairseq/modules/norms/mask_powernorm.py))

## Motivation

MoE as in GShard should be a crucial addition for this library, since the performance gain from it is rather significant as demonstrated in the paper. For example, Transformer+MoE achieved 44.3 avg BLEU on various NMT tasks with 10x less computes than with the baseline Transformer to get 36.9 avg BLEU. The codes for GShard has not been available yet, and if it will be published, it is most likely to be published in Tensorflow (or JAX) rather than PyTorch due to their use of TPUs. However, the codes for MoE does not seem to be complicated, so it must be easy to implement it.

PowerNorm improved Wikitext-103 perplexity from 20.9 to 17.9 without additional computes simply by replacing LayerNorm with PowerNorm. Given that there is already PyTorch codes available, which is easy to transplant into this library, I think it's reasonable to suggest its feature request.

## Your contribution
As for MoE, I'm currently writing my implementation of non-hierarchical MoE in PyTorch (no model parallelism as in GShard for numerous GPUs), and I'm going to compare its performance against the baseline. It should serve as a reference. My collaborator (@lucidrains) may write the same one with more flexibilities later, but no guarantee. You may want hierarchical MoE and GShard components as well. If you want them, my codes may not be as useful.

As for PowerNorm, I don't think you need much help from me, as all you need is to replace LayerNorm with the above implemenation. I'll try to verify its performance gain, since replication of the results hasn't been done yet.


[cleanup] T5 test, warnings

- simplify shape checking, task specific params usage
- add task specific params logger statement to examples
- Let model.config.max_length determine document length for evaluation


Zero shot classification pipeline

This PR adds a pipeline for zero-shot classification using pre-trained NLI models as demonstrated in our [zero-shot topic classification demo](https://huggingface.co/zero-shot/) and [blog post](https://joeddav.github.io/blog/2020/05/29/ZSL.html).

Addresses #5756, where @clmnt requested zero-shot classification in the inference API. However, it should be noted that this model has a max sequence size of `1024`, so long documents would be truncated to this length when classifying.

The pipeline takes in collection of sequences and labels. The labels are converted into a hypothesis, e.g. `vulgar` ➡ `this example is vulgar.` Each sequence and each candidate label must be paired and passed through the model, so the total number of forward passes is `num_labels * num_sequences`.

#### Usage

The pipeline can be initialized using the `pipeline` factory:

```python
from transformers import pipeline
nlp = pipeline("zero-shot-classification")
```
Then any combination of sequences and candidate labels can be passed.
```python
sequence_to_classify = "Who are you voting for in 2020?"
candidate_labels = ["Europe", "public health", "politics"]
nlp(sequence_to_classify, candidate_labels)
>>> {'sequence': 'Who are you voting for in 2020?',
     'labels': ['politics', 'Europe', 'public health'],
     'scores': [0.9676316380500793, 0.019536184147000313, 0.012832209467887878]}
```
When more than one label is passed, we assume that there is only one true label and that the others are false so that the output probabilities add up to 1. This can be changed by passing `multi_class=True`:
```python
sequence_to_classify = "Who are you voting for in 2020?"
candidate_labels = ["Europe", "public health", "politics", "elections"]
nlp(sequence_to_classify, candidate_labels, multi_class=True)
>>> {'sequence': 'Who are you voting for in 2020?',
     'labels': ['politics', 'elections', 'Europe', 'public health'],
     'scores': [0.9720695614814758,
      0.967610776424408,
      0.060417089611291885,
      0.03248738870024681]}
```
The single-label case is likely to be more reliable, however, since the guarantee of only one true label provides strong signal to the model which is very useful in the zero-shot setting.

By default, labels are turned into NLI-format hypotheses with the template `This example is {label}.`. You can change this by with the `hypothesis_template` argument, but the default template seems to work well in most settings I've experimented with.

A couple more examples:
```python
reviews = [
     "I didn't care for this film, but the ending was o.k.",
     "There were some weak moments, but the movie was pretty good overall"
]
nlp(reviews, ["positive", "negative"])
>>> [{'sequence': "I didn't care for this film, but the ending was o.k.",
      'labels': ['negative', 'positive'],
      'scores': [0.9887893199920654, 0.011210653930902481]},
     {'sequence': 'There were some weak moments, but the movie was pretty good overall',
      'labels': ['positive', 'negative'],
      'scores': [0.6071907877922058, 0.3928091824054718]}]
```
```python
reviews = [
     "I didn't care for this film, but the ending was o.k.",
     "There were some weak moments, but the movie was pretty good overall"
]
hypothesis_template = 'The sentiment of this review is {}.'
nlp(reviews, ["positive", "negative"], hypothesis_template=hypothesis_template)
>>> [{'sequence': "I didn't care for this film, but the ending was o.k.",
      'labels': ['negative', 'positive'],
      'scores': [0.9774571061134338, 0.022542938590049744]},
     {'sequence': 'There were some weak moments, but the movie was pretty good overall',
      'labels': ['positive', 'negative'],
      'scores': [0.9787198305130005, 0.021280216053128242]}]
```
```python
nlp("I am a bit discouraged by my grades.", ["sad", "happy", "angry"])
>>> {'sequence': 'I am a bit discouraged by my grades.',
     'labels': ['sad', 'angry', 'happy'],
     'scores': [0.9630885124206543, 0.0311590563505888, 0.005752446595579386]}
```


T5 Model Cards

Related to pretrained model cards

All are identical.
Happy to update w more info!
I didn't have task tags because they seem to already work.


metadata

Related to pretrained model cards


BART/T5 eli5 in model hub

# 🚀 Feature request

Would love to have https://huggingface.co/qa in the model hub mostly for the inference widget/api (the demo is down quite regularly)

## Motivation

A lot of companies might want to test it/use it!

## Your contribution

🔥 emojis on slack


BART MNLI + yahoo answer in the model hub for inference API

# 🚀 Feature request

would love to be able to use https://huggingface.co/zero-shot/ with the inference API

## Motivation

it would help many companies run zero-shot classification of long documents

## Your contribution

I can provide 🔥 emojis on slack


Problems with generating text using mbart-large-cc25

# 🐛 Bug

## Information

Model I am using (Bert, XLNet ...): MBART

Language I am using the model on (English, Chinese ...): English, Romanian

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

I'm examining 'facebook/mbart-large-en-ro' and 'facebook/mbart-large-cc25' checkpoints of MBART.

Here is my first script translating an English sentence to Romanian:

```
from transformers import MBartTokenizer, BartForConditionalGeneration

tokenizer = MBartTokenizer.from_pretrained('facebook/mbart-large-en-ro')
model = BartForConditionalGeneration.from_pretrained('facebook/mbart-large-en-ro')

src_sent = "UN Chief Says There Is No Military Solution in Syria"

src_ids = tokenizer.prepare_translation_batch([src_sent])

output_ids = model.generate(src_ids["input_ids"], decoder_start_token_id=tokenizer.lang_code_to_id["ro_RO"])

output = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0]

print('src_sent: ', src_sent)
print('src_ids: ', src_ids)
print('output_ids: ', output_ids)
print('output: ', output)

```

stdout:

```
src_sent:  UN Chief Says There Is No Military Solution in Syria
src_ids:  {'input_ids': tensor([[  8274, 127873,  25916,      7,   8622,   2071,    438,  67485,     53,
         187895,     23,  51712,      2, 250004]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}
output_ids:  tensor([[250020,      0,  47711,   7844, 127666,      8,  18347,  18147,   1362,
            315,  42071,     36,  31563,   8454,  33796,    451,    346, 125577]])
output:  Şeful ONU declară că nu există o soluţie militară în Siria

```

As seen in output_ids the model always generates 0 (corresponding to bos_token) at the first decoding step. However, this seems to not be a problem with this checkpoint as the output is still the correct translation.

Now I run the same script but using pretrained "facebook/mbart-large-cc25" and trying to denoise an English input. Since the input does not have mask tokens the output should be identical to the input given the pertaining objective of MBART.
However, the output always misses the first token from the input. I have observed this with different examples (even when you have masked tokens in the input).

```
from transformers import MBartTokenizer, BartForConditionalGeneration

tokenizer = MBartTokenizer.from_pretrained('facebook/mbart-large-cc25')
model = BartForConditionalGeneration.from_pretrained('facebook/mbart-large-cc25')

src_sent = "UN Chief Says There Is No Military Solution in Syria"

src_ids = tokenizer.prepare_translation_batch([src_sent])

output_ids = model.generate(src_ids["input_ids"], decoder_start_token_id=tokenizer.lang_code_to_id["en_XX"])

output = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0]

print('src_sent: ', src_sent)
print('src_ids: ', src_ids)
print('output_ids: ', output_ids)
print('output: ', output)

```

the stdout:

```
src_sent:  UN Chief Says There Is No Military Solution in Syria
src_ids:  {'input_ids': tensor([[  8274, 127873,  25916,      7,   8622,   2071,    438,  67485,     53,
         187895,     23,  51712,      2, 250004]]), 'attention_mask': tensor([[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]])}
output_ids:  tensor([[250004,      0, 127873,  25916,      7,   8622,   2071,    438,  67485,
             53, 187895,     23,  51712]])
output:  Chief Says There Is No Military Solution in Syria
```

I have tried various approaches but haven't found any clear solutions to this. Appreciate any help on this.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

<!-- A clear and concise description of what you would expect to happen. -->

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2
- Python version: 3.7.4
- PyTorch version (GPU?): 1.4.0 (False)


T5 fine-tuned model doesn't appear in the model hub

Hi guys,
I have fine-tuned T5-base on wikiSQL dataset and uploaded it to HF model hub. The problem is that the model doesn't appear there. AFAIK it is because the config file is not right. Should I change anything?

Thanks.