# ❓ When I used transformers based on jupyter, it can't get vocab,but  the function AlbertModel.from_pretrained  is available

**CODE:**
from transformers import AlbertTokenizer, AlbertModel
import torch
tokenizer = AlbertTokenizer.from_pretrained("./albert-v1/vocab.txt")

**The following error occurs:**

Calling AlbertTokenizer.from_pretrained() with the path to a single file or url is deprecated
------------------------------------------------------------------
RuntimeError                     Traceback (most recent call last)
<ipython-input-10-bf78623a6e4a> in <module>
----> 1 tokenizer = AlbertTokenizer.from_pretrained("./albert-v1/vocab.txt")

~/anaconda3/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in from_pretrained(cls, *inputs, **kwargs)
   1138
   1139         """
-> 1140         return cls._from_pretrained(*inputs, **kwargs)
   1141
   1142     @classmethod

~/anaconda3/lib/python3.7/site-packages/transformers/tokenization_utils_base.py in _from_pretrained(cls, pretrained_model_name_or_path, *init_inputs, **kwargs)
   1285         # Instantiate tokenizer.
   1286         try:
-> 1287             tokenizer = cls(*init_inputs, **init_kwargs)
   1288         except OSError:
   1289             raise OSError(

~/anaconda3/lib/python3.7/site-packages/transformers/tokenization_albert.py in __init__(self, vocab_file, do_lower_case, remove_space, keep_accents, bos_token, eos_token, unk_token, sep_token, pad_token, cls_token, mask_token, **kwargs)
    153
    154         self.sp_model = spm.SentencePieceProcessor()
--> 155         self.sp_model.Load(vocab_file)
    156
    157     @property

~/anaconda3/lib/python3.7/site-packages/sentencepiece.py in Load(self, model_file, model_proto)
    365       if model_proto:
    366         return self.LoadFromSerializedProto(model_proto)
--> 367       return self.LoadFromFile(model_file)
    368
    369

~/anaconda3/lib/python3.7/site-packages/sentencepiece.py in LoadFromFile(self, arg)
    175
    176     def LoadFromFile(self, arg):
--> 177         return _sentencepiece.SentencePieceProcessor_LoadFromFile(self, arg)
    178
    179     def Init(self,

RuntimeError: Internal: /sentencepiece/src/sentencepiece_processor.cc(818) [model_proto->ParseFromArray(serialized.data(), serialized.size())]

![image](https://user-images.githubusercontent.com/50441790/92065758-cf984780-edd2-11ea-8774-6a9dd5f27b16.png)


T5 Tokenizer fails to decode correctly and prints ⁇

T5 Tokenizer tokenizes this following sequence to
```
>>> from transformers import T5Tokenizer
>>> tokenizer = T5Tokenizer.from_pretrained("t5-base")
>>> print(tokenizer.tokenize("My phone number is  1-${phone.number}"))
['▁My', '▁phone', '▁number', '▁is', '▁1-', '$', '{', 'phone', '.', 'num', 'ber', '}']
```
So far so good but when we decode the above sequence back, we get weird ⁇ symbols.

```
>>> print(tokenizer.decode(tokenizer.encode("My phone number is  1-${phone.number}")))
My phone number is 1-$ ⁇ phone.number ⁇
```

This along with the bug https://github.com/huggingface/transformers/issues/6150 shows that T5 Tokenizer
- Is not cycle consistent
- Ignores multiple whitespaces
##  Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2
- Platform: Linux-4.9.0-12-amd64-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.5.1+cpu (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no
### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 -->
T5: @patrickvonplaten
## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Shown above

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
A cycle consistent T5 Tokenizer that works on a variety of inputs


[model weights caching] model upload doesn't check model weights hash

I have re-uploaded model weights via `transformers-cli upload` and noticed that when I tried to use it - it didn't get re-downloaded, and instead continued to use the cached version.

The problem seems to come from the fact that the other uploaded files haven't changed, only the model weights.

I double checked that the md5sum of the old weights file is different from the new one.

I re-uploaded the whole folder using:

```
transformers-cli upload fsmt-wmt19-en-de
```
If I hunt down the cached files (not an easy task), and delete those, it does re-download the new version.

If I diff the cached weights file and the updated cache file, which gets re-downloaded if I move away the original cached file, they aren't the same.:
```
Binary files
before/d97352d9f1f96ee4c6055f203812035b4597258a837db1f4f0803a2932cc3071.53ce64c7097bfcd85418af04a21b4a897c78c8440de3af078e577727ad9de3a0
and
 after/d97352d9f1f96ee4c6055f203812035b4597258a837db1f4f0803a2932cc3071.53ce64c7097bfcd85418af04a21b4a897c78c8440de3af078e577727ad9de3a0
differ
```
Could we please include the model weights file in the hash calculation?

Thank you.


Fix mixed precision issue in TF DistilBert by removing hard-coded uses of float32.

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #6858


Template updates

When adding the Funnel Transformer, I noticed a few things wrong in the template. This PR fixes those.

- using `transformers.testing_utils` instead of `.utils`
- remove xxx from names in tests (as @patrickvonplaten has done recently on Bert)
- add multiple choice model test
- fix label names in masked lm model
- remove the mention to add to pipelines.py in the checklist since there is nothing to do there


Small bug on website

Hi, I am not sure if this is the correct place to report this but all the web pages of https://huggingface.co/transformers/master/index.html are having some issues with scrolling. Scrolling the main text (right side) also scrolls the table of contents(left part)


batch_encode_plus does not lead to the same predictions as encode_plus

I use batch_encode_plus to speed up the predictions but it leads to different results compared to "encode_plus"
```
tokenizer = AutoTokenizer.from_pretrained("deepset/roberta-base-squad2")
inputs = tokenizer.encode_plus(QA_input['question'], QA_input['context'], padding = True, add_special_tokens=True,

```
and
```
tokenizer = AutoTokenizer.from_pretrained("deepset/roberta-base-squad2")
questions = [(q['question'],q['context']) for q in q_dict]
inputs = tokenizer.batch_encode_plus(questions, padding=True, add_special_tokens=True, return_tensors="pt")

```

Most of the time in the predictions are the same but sometimes they are different:

```
def predict_batch_using_model(model, model_name, q_dict):
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    questions = [(q['question'],q['context']) for q in q_dict]
    inputs = tokenizer.batch_encode_plus(questions, padding=True, add_special_tokens=True, return_tensors="pt")
    logger.debug('inputs batch_encode_plus: %s\n',inputs)

    if torch.cuda.is_available():
        inputs.to('cuda')

    answer_start_scores, answer_end_scores = model(**inputs)

    # a list of (answer, probs_start)
    answer_probs_start_batch = []

    for i in range(len(q_dict)):
        input_ids = inputs["input_ids"].tolist()[i]

        text_tokens = tokenizer.convert_ids_to_tokens(input_ids)

        answer_start = torch.argmax(
            answer_start_scores[i]
        )  # Get the most likely beginning of answer with the argmax of the score
        answer_end = torch.argmax(answer_end_scores[i]) + 1  # Get the most likely end of answer with the argmax of the score
        logger.debug('answer_start, answer_end: %d %d %d\n',i, answer_start, answer_end)
        answer = tokenizer.convert_tokens_to_string(text_tokens[answer_start:answer_end])

        total_scores = answer_start_scores[i].add_(answer_end_scores[i])  # in place addition
        total_scores = total_scores.cpu().data.numpy()

        probs = _compute_softmax(total_scores)

        answer_probs_start_batch.append( ( answer, probs[answer_start]))

    return answer_probs_start_batch

def predict_using_model(model, model_name, QA_input):
    tokenizer = AutoTokenizer.from_pretrained(model_name)

    inputs = tokenizer.encode_plus(QA_input['question'], QA_input['context'], padding = True, add_special_tokens=True, return_tensors="pt")
    logger.debug('inputs encode_plus: %s\n',inputs)

    if torch.cuda.is_available():
        inputs.to('cuda')

    input_ids = inputs["input_ids"].tolist()[0]

    text_tokens = tokenizer.convert_ids_to_tokens(input_ids)
    answer_start_scores, answer_end_scores = model(**inputs)

    answer_start = torch.argmax(
        answer_start_scores
    )  # Get the most likely beginning of answer with the argmax of the score
    answer_end = torch.argmax(answer_end_scores) + 1  # Get the most likely end of answer with the argmax of the score
    logger.debug('answer_start, answer_end: %d %d %d\n',0, answer_start, answer_end)

    answer = tokenizer.convert_tokens_to_string(text_tokens[answer_start:answer_end])

    total_scores = answer_start_scores.add_(answer_end_scores)  # in place addition
    total_scores = total_scores.cpu().data.numpy()

    probs = _compute_softmax(total_scores)

    return answer, probs[answer_start]

```
the input dictionary
`
{"q": "what is color of thomas train", "gt_answer": "blue", "results": [{"system": "Google KG", "response": [], "latency": 350}, {"system": "Google CSE", "response": [{"source": "www.strasburgrailroad.com", "title": "15 Fun Facts About Thomas the Tank Engine - Strasburg Rail Road", "snippet": "Aug 15, 2017 ... Thomas' iconic blue color is also the official color of the North Western Railway. \nBefore Thomas was blue he was originally teal green with\u00a0..."}, {"source": "www.youtube.com", "title": "Learn Colors with My First Railways | Playing Around with Thomas ...", "snippet": "Oct 21, 2017 ... About Thomas & Friends: Based on a series of children's books, \"Thomas & \nFriends\" features Thomas the Tank Engine adventures with other\u00a0..."}, {"source": "en.wikipedia.org", "title": "Thomas the Tank Engine - Wikipedia", "snippet": "Thomas the Tank Engine is an anthropomorphised fictional steam locomotive in \nThe Railway ... In The Adventure Begins which is a retelling of Thomas's early \ndays on Sodor, he is a bluish-green colour when he first arrives on Sodor, his \ntanks\u00a0..."}, {"source": "play.thomasandfriends.com", "title": "Meet the Thomas & Friends Engines | Thomas & Friends", "snippet": "Discover all the engines from Sodor! Thomas & Friends fans can learn about all \ntheir favorite characters from the Thomas & Friends books, TV series and\u00a0..."}, {"source": "www.theguardian.com", "title": "Thomas the Tank Engine had to shut the hell up to save children ...", "snippet": "Jul 22, 2014 ... Thomas the Tank Engine had to shut the hell up to save children everywhere. \nThis article is more than 6 years old. Tracy Van Slyke. Classism\u00a0..."}, {"source": "www.amazon.com", "title": "RoomMates RMK1035SCS Thomas & Friends Peel ... - Amazon.com", "snippet": "RoomMates RMK1035SCS Thomas & Friends Peel and Stick Wall Decals ,Multi \ncolor. +. RoomMates RMK1831SCS Thomas The Tank Engine Peel and Stick\u00a0..."}, {"source": "ttte.fandom.com", "title": "Nia | Thomas the Tank Engine Wikia | Fandom", "snippet": "Nia is a Kenyan tank engine who befriended and accompanied Thomas on his \njourney ... Noticing how heavy his train was getting, she offered to help, but \nThomas ... of the Steam Team to have a snowplough that is not the same colour \nas her."}, {"source": "www.amazon.com", "title": "Thomas The Tank Engine Color Block Cotton Hand ... - Amazon.com", "snippet": "Buy Thomas The Tank Engine Color Block Cotton Hand Towel: Home & Kitchen - \nAmazon.com \u2713 FREE DELIVERY possible on eligible purchases."}, {"source": "ttte.fandom.com", "title": "Thomas the Tank Engine Colors", "snippet": "Thomas the Tank Engine: Colors is a book. Characters Thomas, Edward, Henry, \nJames, Percy, Bill..."}, {"source": "www.pinterest.com", "title": "Train cake, Thomas train cake, Thomas the train", "snippet": "Fondant Train Topper with Mini Train Cupcake Toppers. Each Topper is made to \norder and can be customized to suit your color scheme. Lot comes\u00a0..."}], "latency": 663}, {"system": "Bing entity", "response": [], "latency": 698}, {"system": "Bing web", "response": [{"source": "www.youtube.com", "title": "What Color Was Thomas the Tank Engine? | The Earl's Quiz ...", "snippet": "Based on a series of children's books, \"Thomas & Friends\" features Thomas the Tank Engine adventures with other locomotives on the island of Sodor. Thomas often gets into trouble, but never gives ..."}, {"source": "british-learning.com", "title": "Thomas The Train Color Pages To Print \u2013 Learning How to Read", "snippet": "Thomas and friends coloring pages 55 thomas and friends pictures to print and color. 55 thomas and friends printable coloring pages for kids. 30 free printable thomas the train coloring pages. For boys and girls kids and adults teenagers and toddlers preschoolers and older kids at school."}, {"source": "www.hometalk.com", "title": "Does anybody know what color blue is used for Thomas the ...", "snippet": "Here is a step by step YouTube guide to painting Thomas The Tank Engine and midway through, the blue used is referred to as a medium blue. Lighter than Navy, darker than Sky, maybe like a colonial blue? https://www.youtube.com/watch?v=MU8L6tIHk08"}], "latency": 879}], "dt": "2020-08-14T15:06:39.638346+00:00"}`

  we observe difference in the prediction of the tenth context. Any reason for that?


[s2s]: script to convert pl checkpoints to hf checkpoints


adding additional additional_special_tokens to tokenizer has inconsistent behavior

## Environment info

- `transformers` version: 3.1.0
- Platform: Linux-5.4.0-45-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.6.10
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help

@mfuntowicz

## Information

affected: all tokenizers based on `SpecialTokensMixin`

The behavior of the `add_special_tokens()` method seems irregular to me, when adding `additional_special_tokens` to a tokenizer that already holds a list of `additional_special_tokens`. In this case the value of `self._additional_special_tokens` will simply be replaced, while the previous additional special tokens still remain in `PreTrainedTokenizer.added_tokens_encoder` and `PreTrainedTokenizer.added_tokens_decoder`.

## To reproduce

Steps to reproduce the behavior:

```python
from transformers import GPT2Tokenizer

def print_special_tokens():
    print(tokenizer.all_special_tokens)
    print(tokenizer.all_special_ids)
    print(tokenizer.additional_special_tokens)
    print(tokenizer.additional_special_tokens_ids)
    print(tokenizer.special_tokens_map)
    print(tokenizer.added_tokens_encoder)
    print(tokenizer.added_tokens_decoder)

tokenizer = GPT2Tokenizer.from_pretrained(
    'gpt2',
    pad_token='[PAD]',
    additional_special_tokens=['<A>', '<B>', '<C>']
)
print_special_tokens()
tokenizer.add_special_tokens({
    'cls_token': '[CLS]',
    'additional_special_tokens': ['<B>', '<X>', '<X>']
})
print('-'*50)
print_special_tokens()
```

Output:

```
['<|endoftext|>', '<|endoftext|>', '<|endoftext|>', '[PAD]', '<A>', '<B>', '<C>']
[50256, 50256, 50256, 50257, 50258, 50259, 50260]
['<A>', '<B>', '<C>']
[50258, 50259, 50260]
{'bos_token': '<|endoftext|>', 'eos_token': '<|endoftext|>', 'unk_token': '<|endoftext|>', 'pad_token': '[PAD]', 'additional_special_tokens': "['<A>', '<B>', '<C>']"}
{'[PAD]': 50257, '<A>': 50258, '<B>': 50259, '<C>': 50260}
{50257: '[PAD]', 50258: '<A>', 50259: '<B>', 50260: '<C>'}
--------------------------------------------------
['<|endoftext|>', '<|endoftext|>', '<|endoftext|>', '[PAD]', '[CLS]', '<B>', '<X>']
[50256, 50256, 50256, 50257, 50261, 50259, 50262]
['<B>', '<X>', '<X>']
[50259, 50262, 50262]
{'bos_token': '<|endoftext|>', 'eos_token': '<|endoftext|>', 'unk_token': '<|endoftext|>', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'additional_special_tokens': "['<B>', '<X>', '<X>']"}
{'[PAD]': 50257, '<A>': 50258, '<B>': 50259, '<C>': 50260, '[CLS]': 50261, '<X>': 50262}
{50257: '[PAD]', 50258: '<A>', 50259: '<B>', 50260: '<C>', 50261: '[CLS]', 50262: '<X>'}
```

## Expected behavior

Additional special tokens added by `add_special_tokens()` should be appended to the existing `_additional_special_tokens` list and not replace them. Also, there should be some deduplication logic.

The following code change in `SpecialTokensMixin.add_special_tokens()` will do exactly this:

```python
        for key, value in special_tokens_dict.items():
            assert key in self.SPECIAL_TOKENS_ATTRIBUTES, f"Key {key} is not a special token"

            if key == "additional_special_tokens":
                assert isinstance(value, (list, tuple)) and all(
                    isinstance(t, (str, AddedToken)) for t in value
                ), f"Tokens {value} for key {key} should all be str or AddedToken instances"

                if self.verbose:
                    logger.info("Adding %s to `additional_special_tokens`", value)

                for token in value:
                    if token not in self.additional_special_tokens:
                        self._additional_special_tokens.append(token)

                added_tokens += self.add_tokens(value, special_tokens=True)
            else:
                assert isinstance(
                    value, (str, AddedToken)
                ), f"Token {value} for key {key} should be a str or an AddedToken instance"

                if self.verbose:
                    logger.info("Assigning %s to the %s key of the tokenizer", value, key)
                setattr(self, key, value)

                added_tokens += self.add_tokens([value], special_tokens=True)
```

Now, when running the above code example the output is as expected (imho):

```
['<|endoftext|>', '<|endoftext|>', '<|endoftext|>', '[PAD]', '<A>', '<B>', '<C>']
[50256, 50256, 50256, 50257, 50258, 50259, 50260]
['<A>', '<B>', '<C>']
[50258, 50259, 50260]
{'bos_token': '<|endoftext|>', 'eos_token': '<|endoftext|>', 'unk_token': '<|endoftext|>', 'pad_token': '[PAD]', 'additional_special_tokens': "['<A>', '<B>', '<C>']"}
{'[PAD]': 50257, '<A>': 50258, '<B>': 50259, '<C>': 50260}
{50257: '[PAD]', 50258: '<A>', 50259: '<B>', 50260: '<C>'}
--------------------------------------------------
['<|endoftext|>', '<|endoftext|>', '<|endoftext|>', '[PAD]', '[CLS]', '<A>', '<B>', '<C>', '<X>']
[50256, 50256, 50256, 50257, 50261, 50258, 50259, 50260, 50262]
['<A>', '<B>', '<C>', '<X>']
[50258, 50259, 50260, 50262]
{'bos_token': '<|endoftext|>', 'eos_token': '<|endoftext|>', 'unk_token': '<|endoftext|>', 'pad_token': '[PAD]', 'cls_token': '[CLS]', 'additional_special_tokens': "['<A>', '<B>', '<C>', '<X>']"}
{'[PAD]': 50257, '<A>': 50258, '<B>': 50259, '<C>': 50260, '[CLS]': 50261, '<X>': 50262}
{50257: '[PAD]', 50258: '<A>', 50259: '<B>', 50260: '<C>', 50261: '[CLS]', 50262: '<X>'}
```

I could open a PR if you agree that this is indeed the expected behavior.


[style] automate reformatting with pre-commit hooks

# 🚀 Feature request

I was just reading how `make style` can be automated with pre-commit hooks. Noticing how often I run and even more often forget to run `make style` before committing, perhaps others are in the same boat - and therefore I thought to propose to the dev community to (mostly) automate this process. The only cons is that each dev will still have to run `pre-commit install` once after cloning the project. This is a security feature of git, so it won't run anything automatically unless you take action to enable such thing.

If I understand it correctly, if an individual dev doesn't run `pre-commit install` inside the repo, things are just as normal as they are now. That dev will just run `make style` manually. i.e. the proposed feature is optional for those who want it.

I read about it [here](https://www.mattlayman.com/blog/2018/python-code-black/), please scroll down to the section: "Black as a Git pre-commit hook". And it links to the whole detailed website: https://pre-commit.com/


Funnel transformer

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #4844

This PR adds the Funnel Transformer architecture in PyTorch. For now, I have uploaded two of the ten checkpoints for this model, I will convert and upload the other ones while this PR is under review and add them before it's merged.

Due to the fact there are two versions of the Funnel model (one that returns hidden states with a sequence length divided by 4 and one that return hidden states with the same sequence length, that has 2 more layers), I add to make to different Tester in the test file (because the expected number of hidden states / attentions change depending on which model is used). I adapted the script that check all models are tested to account for that.


Torchscript benchmark measure

This PR is just there to show some benchmarking results of `BertScriptableModel` vs. `BertModel`. It shows the results of running the script: `benchmark_pytorch_scripting.py`.

In a nutshell, the script does the following:

1) Create a list of 500 and 2500 `input_tensors` of `batch_size` 1 with a sequence length varying between 1 and 128 or 1 and 512.
    Then take a scripted model `model = torch.jit.script(BertScriptableModel(...))` and loop over all 500 / 2500 `input_tensors` in a standard for loop. The script model is warmed up by running the loop 5 times before measuring the time. The loop is run 10 times and the fastest run is taken as a measurement.

2) Create a list of 64 and 512 input_tensors of batch_size 8 with a sequence length varying between 1 and 128 or 1 and 512.
    Then take a scripted model `model = torch.jit.script(BertScriptableModel(...))` and loop over all 64 / 512 `input_tensors` in a standard for loop. The script model is warmed up by running the loop 5 times before measuring the time. The loop is run 10 times and the fastest run is taken as a measurement.

All this was done on the following environment information:
```
====================        ENVIRONMENT INFORMATION         ====================
- transformers_version: 3.0.0
- framework: PyTorch
- use_torchscript: True
- framework_version: 1.6.0
- python_version: 3.6.10
- system: Linux
- cpu: x86_64
- architecture: 64bit
- date: 2020-09-02
- time: 16:26:10.562635
- fp16: False
- use_multiprocessing: False
- only_pretrain_model: False
- cpu_ram_mb: 32088
- use_gpu: True
- num_gpus: 1
- gpu: TITAN RTX
- gpu_ram_mb: 24217
- gpu_power_watts: 280.0
- gpu_performance_state: 2
- use_tpu: False
```

=> So only on GPU.

To run this script, one can simply run:

```
./benchmark_pytorch_scripting.py
```

**Important**:

The "for" loop corresponds to the function defined in lines 32 - 37 of the file `benchmark_pytorch_scripting.py`.
This function then overwrites the function that is usually measured in benchmarks, by setting `benchmark._prepare_inference_func = _prepare_inference_func` in line 49.

It would be awesome if @sbrody18 could take a look at the `benchmark_pytorch_scripting.py` f file to check if torchscript was used correctly.


Update to the huBERT model card.

model card

Related to pretrained model cards

Added a link to the thesis.


Changed link to the correct paper in the second paragraph

Related to pretrained model cards


Greedy decoding for non-beam-search appears to ignore postprocessing

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2
- Platform: Linux-4.4.0-18362-Microsoft-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0 (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

TextGeneration: @TevenLeScao

## Information

Model I am using (Bert, XLNet ...): Bart

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

I was experimenting with `generate` method to understand behavior on small cases.

## To reproduce

See https://github.com/huggingface/transformers/blob/8f2723caf0f1bf7e1f639d28d004f81c96d19bbc/src/transformers/generation_utils.py#L535-L566

The last line should probably take `argmax` of post-processed `scores` instead of `next_token_logits`. This should manifest in not respecting minimum length, generating bad words and repeats.

On a more minor note, are `next_token_logscores` really _log_ scores?

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->


Output attention takes an s

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #6902

Stas would have come with a nice Perl magic command but I did a regex search (`output_attention[^s]`) to fix all those misspelled args. In the process, I noticed a few examples were missing a line so added that too.


Example config code uses invalid 'output_attention' rather than 'output_attentions'

Looks like documentation only bug. `output_attention` used rather than `output_attentions`.

Occurs in multiple places in repo.

Maybe linked to #2985

## Environment info

- `transformers` version: 3.1.0
- Platform: Darwin-19.6.0-x86_64-i386-64bit
- Python version: 3.7.4
- PyTorch version (GPU?): 1.5.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
 documentation: @sgugger

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

From docs L4-L5 of AutoConfig example:
[docs](https://huggingface.co/transformers/model_doc/auto.html#transformers.AutoConfig)
or
[code](https://github.com/huggingface/transformers/blob/master/src/transformers/configuration_auto.py#L264)

```python
config = AutoConfig.from_pretrained('bert-base-uncased', output_attention=True, foo=False)
assert config.output_attention == True
```

causes:

```
AttributeError: 'BertConfig' object has no attribute 'output_attention'
```

## Expected behavior

The assertion given in documentation passes.


Relaxing `PreTrainedModel` requirement in _save

# 🚀 Feature request
It's great to see that `Trainer` is becoming flexible. Each functions seems to be more self contained now making inheritance easier. I've experimented with many custom models. For instance,
```
class Model(nn.Module):
    def __init__(self, ..):
        self.encoder = AutoModel.from_pretrained(..)
        self.custom_modules = ..
    def forward(self, **kwargs):
        output = self.encoder(**kwargs)
        # some custom operations
```
Many users are required to create custom models if they just don't want simple `SequenceClassification` head. In all cases, I have to override `_save` method because of [this line](https://github.com/huggingface/transformers/blob/d822ab636b6a14ed50f7bca0797c1de42c19de61/src/transformers/trainer.py#L1097) which explicitly puts a restriction on `Trainer` to be used with models that inherit from `PreTrainedModel`. It would be good to relax this requirement and give a warning about not using `PreTrainedModel` instead.

## Your contribution
I'll open a PR if I get approval.


Can DistilBert.forward() support token_type_ids ?

I am using DistilBert to distill a pretrained Bert model. That is Bert -> DistilBert.
The input of Bert is a sentence pair: [CLS] Hello word [SEP] Hello Python [SEP].
But DistilBert dose not support sentence pair inputs.
Can DistilBert support sentence pair-like inputs?


Can the GPT2 of Transformers receive output hidden_states from external Encoder?

# ❓ Questions & Help
<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
I want to use the GPT2 receive the output hidden_states from bert to calculate self_attention ,how can I do?
Thanks.
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


[testing] fix ambiguous test

Since `generate()` does:
```
        num_beams = num_beams if num_beams is not None else self.config.num_beams
```
This test fails if `model.config.num_beams > 1` (which is the case in the model I'm porting).

This fix makes the test setup unambiguous by passing an explicit `num_beams=1` to `generate()`.

Thanks.


Update modeling_bert.py

outptus -> outputs in example of BertForPreTraining

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #{issue number}


the result of translation task on en-zh is not good,especially in short text

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


Create README.md

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #{issue number}


Getting import error

```
from transformers import BertPreTrainedModel, RobertaModel
import torch

class RobertaForMD(BertPreTrainedModel): # Metaphor Detection, modified from BertForTokenClassification
    def __init__(self, config):
        super().__init__(config)
        self.num_labels = config.num_labels

        self.bert = RobertaModel(config)
        self.dropout = torch.nn.Dropout(config.hidden_dropout_prob)
        self.classifier = torch.nn.Linear(config.hidden_size, self.config.num_labels)
        # self.loss = torch.nn.BCEWithLogitsLoss(pos_weight=torch.tensor([3], dtype=torch.float32))
        self.loss = torch.nn.BCEWithLogitsLoss()

        self.init_weights()

    def forward(
        self,
        input_ids=None,
        attention_mask=None,
        token_type_ids=None,
        position_ids=None,
        head_mask=None,
        inputs_embeds=None,
        labels=None,
        word_posi=None
    ):
        outputs = self.bert(
            input_ids,
            attention_mask=attention_mask,
            token_type_ids=token_type_ids,
            position_ids=position_ids,
            head_mask=head_mask,
            inputs_embeds=inputs_embeds,
        )

        last_hidden_state = outputs[0]
        last_hidden_state = self.dropout(last_hidden_state)
        batch_size = input_ids.shape[0]

        word_state = torch.empty((0, last_hidden_state.shape[2]), dtype=torch.float32).cuda()
        for i in range(batch_size):
            word_state = torch.cat((word_state, last_hidden_state[i][word_posi[i]].unsqueeze(0)))

        logits = self.classifier(word_state)

        outputs = (logits,) + outputs[2:]  # add hidden states and attention if they are here

        if labels is not None:
            loss = self.loss(logits.view(-1), labels.to(torch.float32))
            outputs = (loss,) + outputs

        return outputs  # (loss), logits, (hidden_states), (attentions)
```

I am calling this using
model = RobertaForMD.from_pretrained(model_name, num_labels=1)

Name: transformers
Version: 2.7.0

 File "main.py", line 276, in main
    model = RobertaForMD.from_pretrained(model_name, num_labels=1)
  File "/nas/home/tuhinc/miniconda3/lib/python3.7/site-packages/transformers/modeling_utils.py", line 438, in from_pretrained
    **kwargs,
  File "/nas/home/tuhinc/miniconda3/lib/python3.7/site-packages/transformers/configuration_utils.py", line 199, in from_pretrained
    config_dict, kwargs = cls.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/nas/home/tuhinc/miniconda3/lib/python3.7/site-packages/transformers/configuration_utils.py", line 269, in get_config_dict
    raise EnvironmentError(msg)
OSError: Can't load 'roberta-large'. Make sure that:

- 'roberta-large' is a correct model identifier listed on 'https://huggingface.co/models'

- or 'roberta-large' is the correct path to a directory containing a 'config.json' file


Model card for huBERT

Related to pretrained model cards


[t5] Missing requirements in examples/seq2seq

Hi! This is a very small fix. But it seems like some requirements for examples/seq2seq are missing? Namely, rouge-score, gitpython, sacrebleu. Is this intentional (conflicts with other example requirements)? Of course, would be happy to open a PR

## Environment info

- `transformers` version: 8b884dadc6bd70600c98bb35b522beb0005e3f28
- Platform: Linux
- Python version: 3.8
- PyTorch version (GPU?): GPU, '1.6.0+cu101'
- Tensorflow version (GPU?): n/a
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

 Summarization: @sshleifer

## Information

Model I am using (Bert, XLNet ...): T5

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name) CNN Summary
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Create new `conda` env
2. Install requirements.txt in /examples
3. Try running finetune_t5.sh

## Expected behavior

Finetune should run correctly.


## Environment info
- `transformers` version: 3.1.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?: <True>
- Using distributed or parallel set-up in script?: <False>

### Who can help
 Trainer: @sgugger
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 Bart: @sshleifer
 examples/bert-loses-patience: @JetRunner
 examples/token-classification: @stefan-it

## Information

Model I am using (Bert, XLNet ...):

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

```python
def flair_lstm(text):
    sentence = flair.data.Sentence(text)
    flair_sent.predict(sentences=sentence)
    total_sent = sentence.labels
    for label in total_sent:
        value = label.value
        score = label.score
        return '1' if value == 'POSITIVE' else '-1'
```

The tasks I am working on is:
* I'm working with flair to get classification polarities, but the issue seems refer to transformers

## To reproduce

Steps to reproduce the behavior:

1. write
```python
import flair
flair_sent = flair.models.TextClassifier.load('en-sentiment')

def flair_lstm(text):
    sentence = flair.data.Sentence(text)
    flair_sent.predict(sentences=sentence)
    total_sent = sentence.labels
    for label in total_sent:
        value = label.value
        score = label.score
        return '1' if value == 'POSITIVE' else '-1'

df_test = "some test dataframe"
df_test['flair'] = df_test['word'].apply(lambda x: flair_lstm(x))
```
2. See error:

## Traceback:

```python
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-26-1ee39d7138b3> in <module>()
----> 1 df_test['flair'] = df_test['word'].apply(lambda x: flair_lstm(x))

10 frames
pandas/_libs/lib.pyx in pandas._libs.lib.map_infer()

/usr/local/lib/python3.6/dist-packages/transformers/configuration_utils.py in use_return_dict(self)
    217         """
    218         # If torchscript is set, force `return_dict=False` to avoid jit errors
--> 219         return self.return_dict and not self.torchscript
    220
    221     @property

AttributeError: 'DistilBertConfig' object has no attribute 'return_dict'
```


[Docs, Examples] Fix QA example for PT

Fixes #6738.

@sgugger - PyTorch QA example was wrong IMO.


minor docs grammar fixes

just some minor document edits


Create README.md

Related to pretrained model cards

Add language meta attribute

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #{issue number}


Create README.md

Related to pretrained model cards


Create README.md

Related to pretrained model cards

<!-- This line specifies which issue to close after the pull request is merged. -->
model card for  akhooli/xlm-r-large-arabic-sent


Create README.md

Related to pretrained model cards

<!-- This line specifies which issue to close after the pull request is merged. -->
Model card for akhooli/mbart-large-cc25-en-ar


[Electra] fix warning for position ids

<!-- This line specifies which issue to close after the pull request is merged. -->
~Fixes 6882~ (might only be partly)


Create README.md

Adds model card for Longformer2Roberta


Bert Checkpoint Breaks 3.02 -> 3.1.0 due to new buffer in BertEmbeddings

Hi,

Thanks for the great library. I noticed this line being added (https://github.com/huggingface/transformers/blob/v3.1.0/src/transformers/modeling_bert.py#L190) in the latest update.

It breaks checkpoints that were saved when this line wasn't there.

```
	Missing key(s) in state_dict: "generator_model.electra.embeddings.position_ids", "discriminator_model.electra.embeddings.position_ids".
```


'BertEmbeddings' object has no attribute 'bias' while converting tf checkpoint

When trying to convert the checkpoint of a self pre-trained tensorflow BERT model (using the **[create-pretraining.py][1]** script from google) into a pytorch model using **convert_bert_original_tf_checkpoint_to_pytorch.py**

I always end up with the following error:
**AttributeError: 'BertEmbeddings' object has no attribute 'bias'**

The init_vars names (just the first ones) look like this:

```
['bert/embeddings/layer_normalization/beta', 'bert/embeddings/layer_normalization/beta/adam_m', 'bert/embeddings/layer_normalization/beta/adam_v', 'bert/embeddings/layer_normalization/gamma', 'bert/embeddings/layer_normalization/gamma/adam_m', 'bert/embeddings/layer_normalization/gamma/adam_v']
```
Code that produces the error looks like this:
```
for m_name in name:
            if re.fullmatch(r"[A-Za-z]+_\d+", m_name):
                scope_names = re.split(r"_(\d+)", m_name)
            else:
                scope_names = [m_name]
            if scope_names[0] == "kernel" or scope_names[0] == "gamma":
                pointer = getattr(pointer, "weight")
            elif scope_names[0] == "output_bias" or scope_names[0] == "beta":
                print(scope_names)
                pointer = getattr(pointer, "bias")
            elif scope_names[0] == "output_weights":
                pointer = getattr(pointer, "weight")
            elif scope_names[0] == "squad":
                pointer = getattr(pointer, "classifier")
            else:
                try:
                    pointer = getattr(pointer, scope_names[0])
                except AttributeError:
                    logger.info("Skipping {}".format("/".join(name)))
```
Going through all the names and getting the right attributes from the model. When it comes to the Layer Normalization in the BertEmbeddings the script produces an error. Did anyone else encouter that error before? How did you fix this? Did I mix something up with the tensorflow versions? Thanks for your help in advance!

Here again the whole stacktrace:
```
Traceback (most recent call last):
  File "convert_bert_original_tf_checkpoint_to_pytorch.py", line 62, in <module>
    convert_tf_checkpoint_to_pytorch(args.tf_checkpoint_path, args.bert_config_file, args.pytorch_dump_path)
  File "convert_bert_original_tf_checkpoint_to_pytorch.py", line 37, in convert_tf_checkpoint_to_pytorch
    load_tf_weights_in_bert(model, config, tf_checkpoint_path)
  File "/modeling_bert.py", line 136, in load_tf_weights_in_bert
    pointer = getattr(pointer, "bias")
  File "module.py", line 594, in __getattr__
    type(self).__name__, name))
AttributeError: 'BertEmbeddings' object has no attribute 'bias'
```
Bert Config is the following:

```
Building PyTorch model from configuration: BertConfig {
  "attention_probs_dropout_prob": 0.1,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout_prob": 0.1,
  "hidden_size": 512,
  "initializer_range": 0.02,
  "intermediate_size": 2048,
  "layer_norm_eps": 1e-12,
  "max_position_embeddings": 512,
  "model_type": "bert",
  "num_attention_heads": 8,
  "num_hidden_layers": 8,
  "pad_token_id": 0,
  "type_vocab_size": 2,
  "vocab_size": 30522
}
```

  [1]: https://github.com/google-research/bert/blob/master/run_pretraining.py


Fix TF Trainer for TPU

This PR try to fix the trainer for TensorFlow by updating the order of some steps:
1. Dataset preprocessing
2. Strategy creation
3. Model creation

Instead of

1. Strategy creation
2. Model creation
3. Dataset preprocessing

Fixes #6672


Add cache_dir to save features TextDataset

This is in case the dataset is in a RO filesystem, for which is the case
in tests (GKE TPU tests).


[EncoderDecoder] Add xlm-roberta to encoder decoder

This PR adds `XLM-Roberta` to the EncoderDecoder framework by adding a new `XLMRobertaForCausalLM` to the models.

The XLM-Roberta EncoderDecoder can be used as follows:

```python
from transformers import EncoderDecoderModel
import torch

model = EncoderDecoderModel.from_encoder_decoder_pretrained("xlm-roberta-base", "xlm-roberta-base")
input_ids = torch.tensor([10 * [0]])
outputs = model(input_ids, decoder_input_ids=input_ids, labels=input_ids, return_dict=True)
print("Loss", outputs.loss)
```


[WIP, TF] replace keras dense by keras.layers.DenseEinsum

Fixes #6771.

This PR might speed up TF runtime on GPU.
Note that the change requires TF 2.3.0


[TF T5] Possible Error using TF T5 with Keras

See: https://discuss.huggingface.co/t/how-to-train-tft5forconditionalgeneration-model/888.


Restore PaddingStrategy.MAX_LENGTH on QAPipeline while no v2.

QA Pipeline will be slower but will work in all situations.

Need to shift towards pipeline v2 with priority on QA Pipeline to provide a workaround.

Signed-off-by: Morgan Funtowicz <funtowiczmo@gmail.com>


gradient_accumulation_steps in trainer_tf

    part-1:
    self.total_train_batch_size = self.args.train_batch_size * self.args.gradient_accumulation_steps
    ds = (
            self.train_dataset.repeat()
            .shuffle(self.num_train_examples, seed=self.args.seed)
            .batch(self.total_train_batch_size, drop_remainder=self.args.dataloader_drop_last)
            .prefetch(tf.data.experimental.AUTOTUNE)
        )
     part-2:
     for _ in tf.range(self.args.gradient_accumulation_steps):
                reduced_features = {
                    k: ft[: self.args.train_batch_size // self.args.n_replicas] for k, ft in features.items()
                }
                reduced_labels = labels[: self.args.train_batch_size // self.args.n_replicas]

                self.training_step(reduced_features, reduced_labels)

                features = {
                    k: tf.concat(
                        [ft[self.args.train_batch_size // self.args.n_replicas :], reduced_features[k]],
                        axis=0,
                    )
                    for k, ft in features.items()
                }

                labels = tf.concat(
                    [labels[self.args.train_batch_size // self.args.n_replicas :], reduced_labels], axis=0
                )

the implementation of gradient_accumulation seems not friendly to users who have small gpu memory.


Memory blowup with TPU Trainer in master

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 3.0.2   (master)
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.7.0a0+8fb7c50 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?:Yes, TPU v2-8

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 -->
@sgugger @sshleifer @patrickvonplaten
## Information

Recent changes to the Trainer for TPU has resulted in memory blowup during training.
On a machine with 208GB of RAM [sic], this was the memory profile with the master branch on 20th August.
![image](https://user-images.githubusercontent.com/1271289/91822376-90a89d00-ebec-11ea-9c95-948ea93d7e41.png)

This only has increase in memory during evaluation (which is another memory leak bug https://github.com/huggingface/transformers/issues/5509). If you throw enough RAM to the problem, it stays in control.

After the recent changes the memory profile has become this.
![image](https://user-images.githubusercontent.com/1271289/91826302-390d3000-ebf2-11ea-8dce-cd70b49f6ba7.png)

Look how quickly the memory blows up even on this huge machine. I have implemented some optimizations to save memory where I am caching only a single copy of features on redis-server but that is not enough now. The most interesting thing to see is that now the memory also increases during training and not just evaluation.

After these changes, Trainer for TPUs has become unusable for training any practical model and I request you to please look into fixing this.
Model I am using (Bert, XLNet ...): T5

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

Use the TPU example run_language_modelling to reproduce.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
Memory stays constant with the number of training and evaluation iterations.
<!-- A clear and concise description of what you would expect to happen. -->


transformer multitasking

Doing multitasking using transformers library such as this one https://github.com/JayYip/bert-multitask-learning. cws|NER|weibo_ner&weibo_cws, one problem will be sampled at each turn, say weibo_ner&weibo_cws, then weibo_ner and weibo_cws will trained for this turn together.


Albert loads model on both CPU and GPU at the same time

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
MODEL_DEVICE = torch.device('cuda')
MODEL_PATH='./models'
tokenizer = AlbertTokenizer.from_pretrained(MODEL_PATH)
qa_model = AlbertForQuestionAnswering.from_pretrained(MODEL_PATH).to(MODEL_DEVICE)

I am using the above code to load the model.
This model occupies both RAM memory(1.5 GB) and GPU memory(650 MB).
I have specified torch device as Cuda, But still, it doesn't behave as expected.
When "cpu" is specified it works well and doesn't load into GPU. But when cuda is specified it load in CPU and GPU as well.
I  tried , "cuda", "cuda:0"

Any solution to this ?


Updata tokenization_auto.py

Updata tokenization_auto.py to handle Inherited config class.
If ConfigB is inherited from ConfigA, then isinstance(ConfigB(), ConfigA) is true. We hope to use TokenizerB, but TokenizerA will be used incorrectly.


How does relative distance is computed for cross-attention in T5 model?

Let's assume we have a source sequence of length 7 and a target sequence of length 5. In cross-attention sublayer at each decoder layer, every token in the target sequence attends every token in the input sequence.

In T5 model, we compute the relative distance to compute bias using query-len and key-len as in https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_t5.py#L289.

My question is, how the distance between two tokens are computed if they belong to the source and target sequence. The relative distance matrix (5 x 7) would look like:

```
tensor([[ 0,  1,  2,  3,  4,  5,  6],
        [-1,  0,  1,  2,  3,  4,  5],
        [-2, -1,  0,  1,  2,  3,  4],
        [-3, -2, -1,  0,  1,  2,  3],
        [-4, -3, -2, -1,  0,  1,  2]])
```

Once we put the distances into bucket for the cross-attention, it would look like:

```
tensor([[0, 0, 0, 0, 0, 0, 0],
        [1, 0, 0, 0, 0, 0, 0],
        [2, 1, 0, 0, 0, 0, 0],
        [3, 2, 1, 0, 0, 0, 0],
        [4, 3, 2, 1, 0, 0, 0]])
```

Given that cross-attention is a part of the decoder, the `bidirectional` flag is set to False. So, it means while decoding at step `i`, the decoder will treat all the source tokens at position `i, i+1, i+2, ...` having a distance `0` from the target token at position `i`. Is this correct?


MarianMTModel.generate error: Segmentation fault (core dumped)

When I use MarianMTModel to generate sequences, that is running "translated = model.generate(**tokenizer.prepare_seq2seq_batch(src_text))", the python development environment will quit automatically. There is a message, i. e. Segmentation fault (core dumped).
When I run the program in Jupyter, it will also quit. Leave the message like this, "Kernel restarting The kernel appears to have died. It will restart automatically"


[doc] typos

fixed typos


test_tf_common: remove un_used mixin class parameters

Fixes #6590

@sshleifer Is that all for this issue? are there any other cleaning left which I did not understand ❗

Thanks


Is it possible to finetune reformer model for summarization task?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


How to save the whole model as SavedModel format for inference?

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
Hi, I want to save the mode as SavedModel format using model.save(), but when i load it, the input_format is fixed and only input_ids can be used, how can i pass the inputs like {'input_dis':XX, 'attention_mask':XX}?
codes:
![image](https://user-images.githubusercontent.com/38073340/91794928-df871e80-ec4e-11ea-86a8-9c264913b5b7.png)
and then it reports:
![image](https://user-images.githubusercontent.com/38073340/91794969-f168c180-ec4e-11ea-98e1-2e1db94632ea.png)


special token inconsistency for [UNK] token

## Environment info

- `transformers` version: 3.0.2
- Platform: Linux-4.15.0-108-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9

### Who can help
 tokenizers: @mfuntowicz

## Information

Bert tokenizer treats [UNK] token as **special** during `tokenizer.convert_ids_to_tokens(...,skip_special_tokens=True) `
while `tokenizer(...,return_special_tokens_mask=True)` doesn't (for obvious reasons).
I think it would be better to preserve [UNK] tokens in `convert_ids_to_tokens` for consistency of the term "**special token**".

## To reproduce

```python
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
# fast tokenizer also has the same problem
# tokenizer = AutoTokenizer.from_pretrained("bert-large-cased",use_fast=True)
text="sentence 한 셔 word"
print('tokens:',tokenizer.tokenize(text))

tokens=tokenizer(text,return_special_tokens_mask=True)
print("input_ids:",tokens["input_ids"])
print("special_token_mask:",tokens["special_tokens_mask"])
no_special=tokenizer.convert_ids_to_tokens(tokens["input_ids"], skip_special_tokens=True)
special=tokenizer.convert_ids_to_tokens(tokens["input_ids"])
print('tokens from ids (skip special): ',no_special)
print('tokens from ids (skip special): ',special)

print('special tokens',tokenizer.all_special_tokens)
```

## Expected behavior

Also I think keeping [UNK] would be a better behavior, as `convert_ids_to_tokens` is used in inference pipelines and using skip_special_token for getting rid of [CLS][SEP][PAD] tokens leads to unintended loss of [UNK] tokens, which is important.

Related: https://github.com/huggingface/transformers/issues/4391


cleanup: fix typo chunk_size_feed_forward in configuration_utils.py

Problem:

```python
self.chunk_size_feed_forward = kwargs.pop("chunk_size_feed_forward", 0) # line 178
self.chunk_size_feed_forward = kwargs.pop("chunk_size_feed_forwar", 0)# line 198
```
in https://github.com/huggingface/transformers/blob/master/src/transformers/configuration_utils.py#L178

Solution:
delete L198


add a final report to all pytest jobs

we had it added for one job (run_examples_torch), please add it to all pytest jobs - we need the output of what tests were run to debug the codecov issue. thank you!

To remind  `pytest -rA` finalizes the test run with a report like this:
```
PASSED examples/seq2seq/test_seq2seq_examples.py::test_seq2seq_dataset_truncation[patrickvonplaten/t5-tiny-random]
PASSED examples/seq2seq/test_seq2seq_examples.py::test_seq2seq_dataset_truncation[sshleifer/bart-tiny-random]
PASSED examples/seq2seq/test_seq2seq_examples.py::test_seq2seq_dataset_truncation[google/pegasus-xsum]
PASSED examples/seq2seq/test_seq2seq_examples.py::test_legacy_dataset_truncation[sshleifer/bart-tiny-random]
PASSED examples/seq2seq/test_seq2seq_examples.py::test_legacy_dataset_truncation[bert-base-cased]
PASSED examples/test_examples.py::ExamplesTests::test_run_language_modeling
PASSED examples/test_examples.py::ExamplesTests::test_run_pl_glue
PASSED examples/test_examples.py::ExamplesTests::test_run_squad
PASSED examples/bert-loses-patience/test_run_glue_with_pabee.py::PabeeTests::test_run_glue
SKIPPED [1] examples/seq2seq/test_bash_script.py:25: too slow to run on CPU
SKIPPED [1] examples/seq2seq/test_bash_script.py:32: too slow to run on CPU
```


Support nested data structures for input data

# 🚀 Feature request

Support in `train.py` for more complicated/nested data inputs rather than the current assumed single-layer dictionary structure.

## Motivation

In many applications -- for example, mine, where I wish to implement a semi-supervised learning protocol -- the training input might need to be different than the 1-layer dictionary that `train.py` is currently hard-coded to accept.

For instance, in my case, I need a nested structure that supports:

```
    {'supervised_data': {'input_ids':[....], 'labels': [...], 'attention_mask'},
      'unsupervised_data': {'input_ids':[...], 'attention_mask'},
      'augmented_data': {'input_ids':[...], 'attention_mask'}
```
(I am attempting to implement the following paper, Unsupervised Data Augmentation for Consistency Training https://arxiv.org/abs/1904.12848).

However, I can imagine other use-cases including MAML, multi-task learning and multi-modal learning where huggingface would provide a great framework but is currently limited in it's data input format.

## Your contribution

I've identified a couple of quick fixes for this:

Lines 962-974 of `train.py`, or the `_prepare_inputs` function should be rewritten as:

```
    def _prepare_inputs(self, inputs):
        """
        Prepare :obj:`inputs` before feeding them to the model, converting them to tensors if they are not already and
        handling potential state.
        """
        def map_nested_dicts_modify(ob, func):
            if isinstance(ob, dict):
                return {k: map_nested_dicts_modify(v, func) for k, v in ob.items()}
            if isinstance(ob, list):
                return list(map(lambda x: map_nested_dicts_modify(x, func), ob))
            else:
                return func(ob)

        def to_device(v):
            if isinstance(v, torch.Tensor):
                v = v.to(self.args.device)
            return v

        inputs = map_nested_dicts_modify(inputs, to_device)

        if self.args.past_index >= 0 and self._past is not None:
            inputs["mems"] = self._past

        return inputs
```

Line 1135 of `train.py` should be expanded to:

```
        def _finditem(obj, key):
            if key in obj: return True
            for k, v in obj.items():
                if isinstance(v, dict):
                    return _finditem(v, key)

        has_labels = any(_finditem(inputs, k) is not None for k in ["labels", "lm_labels", "masked_lm_labels"])
```

I'm not quite sure how to handle line 1254, or whether it is really that necessary, but one way might be to again use `_finditem` for `["labels", "input_ids"]`.

And that's it -- it's then up to the user to modify their models so that their `DataCollator` generates the expected structure, and `forward` takes in the expected structure!

Even if you explicitly don't wish to support these different training protocols mentioned above, I do think, from a software engineering perspective, that `train.py` should be more fully abstracted from particulars of the data inputs and model inputs. This feature takes you a step closer to that (although not completely, as the `has_labels` line still expects a certain set of keys _somewhere_ in the data input.)

Better yet than the suggestions here would be to make the data input it's own class for full abstraction, but I can see the argument against you doing that, as it is yet another data class for users to learn and code up, and would be a breaking update for all those who have implemented `DataCollators` that adhere to your guidelines.

Alex


[fix] typo in available in helper function

cc @sgugger will merge on ci passing.


Remove hard-coded uses of float32 to fix mixed precision use in Distilbert

In this commit [Remove hard-coded uses of float32 to fix mixed precision use](https://github.com/huggingface/transformers/commit/4fca874ea995f3d23ad7062b07b5ed7c4f87c0cd#diff-e3ab4f29f29fe1d243a6b55fafaab097), the mixed precision issue is fixed for [modeling_tf_bert.py](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_tf_bert.py).

However, for [modeling_tf_distilbert.py](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_tf_distilbert.py#L171), the line 171 is not fixed yet, and we get

>    173         embeddings = inputs_embeds + position_embeddings  # (bs, max_seq_length, dim)
>    --> 174         embeddings = self.LayerNorm(embeddings)  # (bs, max_seq_length, dim)
>    InvalidArgumentError: cannot compute AddV2 as input #1(zero-based) was expected to be a bfloat16 tensor but is a float tensor

while using `mixed_bfloat16 mixed precision` with `TPU`.

A very quick fix is the same as the fix for `modeling_tf_bert.py`:

    position_embeddings = tf.cast(self.position_embeddings(position_ids), inputs_embeds.dtype)

@schmidek


Split hp search methods

Follow-up from #6747.

Cleanly separates the two backend-specific code (optuna vs Ray) with some small code duplication in the objective function.


Changes in Pytorch 1.6 multinomial could break backward compatibility

## Environment info

- `transformers` version: 3.0.2
- Platform: Linux-5.4.0-42-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.6.10
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False

### Who can help

@TevenLeScao

## Information

Model I am using:

Gpt2, but relevant to all language models using sampling.

This is not really a bug report on behalf of transformers, but rather a suggestion to handle a breaking change in the behavior of `torch.multinomial()` that was introduced with v1.6. I've noticed, that the sampling in `GenerationMixin._generate_no_beam_search()` has changed between PT1.5.1 and PT1.6, even when given the same inputs and random state. This is due to a new implementation path of `torch.multinomial()` with `replacement=False`. This new implementation breaks determinism compared to older PT versions. However, there is an easy fix to return to the previous behavior in order to maintain backward compatibility. Since `num_samples` is set to 1, the actual `replacement` value is irrelevant. But setting `replacement` to `True` will use the old sampling implementation and return deterministic results equal to those of earlier versions.

I would therefore recommend to change the call to `multinomial()` in `_generate_no_beam_search()` like this:

```next_token = torch.multinomial(probs, num_samples=1, replacement=True).squeeze(1)```

Best regards,
Andreas


Hugging face - RuntimeError: Caught RuntimeError in replica 0 on device 0 on Azure Databricks

How do I run the run_language_modeling.py script from hugging face using the pretrained roberta case model to fine-tune  using my own data on the Azure databricks with a GPU cluster.

Using Transformer version 2.9.1 and 3.0 .
Python 3.6
Torch `1.5.0
torchvision 0.6

This is the script I ran below on Azure databricks

`
%run '/dbfs/FileStore/tables/dev/run_language_modeling.py' \
  --output_dir='/dbfs/FileStore/tables/final_train/models/roberta_base_reduce_n' \
  --model_type=roberta \
  --model_name_or_path=roberta-base \
  --do_train \
  --num_train_epochs 5 \
  --train_data_file='/dbfs/FileStore/tables/final_train/train_data/all_data_desc_list_full.txt' \
  --mlm
`
This is the error I get after running the above command.

`
 RuntimeError                              Traceback (most recent call last)
/dbfs/FileStore/tables/dev/run_language_modeling.py in <module>
    279
    280 if __name__ == "__main__":
--> 281     main()

/dbfs/FileStore/tables/dev/run_language_modeling.py in main()
    243             else None
    244         )
--> 245         trainer.train(model_path=model_path)
    246         trainer.save_model()
    247         # For convenience, we also re-save the tokenizer to the same directory,

/databricks/python/lib/python3.7/site-packages/transformers/trainer.py in train(self, model_path)
    497                     continue
    498
--> 499                 tr_loss += self._training_step(model, inputs, optimizer)
    500
    501                 if (step + 1) % self.args.gradient_accumulation_steps == 0 or (

/databricks/python/lib/python3.7/site-packages/transformers/trainer.py in _training_step(self, model, inputs, optimizer)
    620             inputs["mems"] = self._past
    621
--> 622         outputs = model(**inputs)
    623         loss = outputs[0]  # model outputs are always tuple in transformers (see doc)
    624

/databricks/python/lib/python3.7/site-packages/torch/nn/modules/module.py in __call__(self, *input, **kwargs)
    548             result = self._slow_forward(*input, **kwargs)
    549         else:
--> 550             result = self.forward(*input, **kwargs)
    551         for hook in self._forward_hooks.values():
    552             hook_result = hook(self, input, result)

/databricks/python/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py in forward(self, *inputs, **kwargs)
    153             return self.module(*inputs[0], **kwargs[0])
    154         replicas = self.replicate(self.module, self.device_ids[:len(inputs)])
--> 155         outputs = self.parallel_apply(replicas, inputs, kwargs)
    156         return self.gather(outputs, self.output_device)
    157

/databricks/python/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py in parallel_apply(self, replicas, inputs, kwargs)
    163
    164     def parallel_apply(self, replicas, inputs, kwargs):
--> 165         return parallel_apply(replicas, inputs, kwargs, self.device_ids[:len(replicas)])
    166
    167     def gather(self, outputs, output_device):

/databricks/python/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py in parallel_apply(modules, inputs, kwargs_tup, devices)
     83         output = results[i]
     84         if isinstance(output, ExceptionWrapper):
---> 85             output.reraise()
     86         outputs.append(output)
     87     return outputs

/databricks/python/lib/python3.7/site-packages/torch/_utils.py in reraise(self)
    393             # (https://bugs.python.org/issue2651), so we work around it.
    394             msg = KeyErrorMessage(msg)
--> 395         raise self.exc_type(msg)

RuntimeError: Caught RuntimeError in replica 0 on device 0.
Original Traceback (most recent call last):
  File "/databricks/python/lib/python3.7/site-packages/torch/nn/parallel/parallel_apply.py", line 60, in _worker
    output = module(*input, **kwargs)
  File "/databricks/python/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/databricks/python/lib/python3.7/site-packages/transformers/modeling_roberta.py", line 239, in forward
    output_hidden_states=output_hidden_states,
  File "/databricks/python/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/databricks/python/lib/python3.7/site-packages/transformers/modeling_bert.py", line 762, in forward
    output_hidden_states=output_hidden_states,
  File "/databricks/python/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/databricks/python/lib/python3.7/site-packages/transformers/modeling_bert.py", line 439, in forward
    output_attentions,
  File "/databricks/python/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/databricks/python/lib/python3.7/site-packages/transformers/modeling_bert.py", line 371, in forward
    hidden_states, attention_mask, head_mask, output_attentions=output_attentions,
  File "/databricks/python/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/databricks/python/lib/python3.7/site-packages/transformers/modeling_bert.py", line 315, in forward
    hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, output_attentions,
  File "/databricks/python/lib/python3.7/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/databricks/python/lib/python3.7/site-packages/transformers/modeling_bert.py", line 240, in forward
    attention_scores = attention_scores / math.sqrt(self.attention_head_size)
RuntimeError: CUDA out of memory. Tried to allocate 96.00 MiB (GPU 0; 11.17 GiB total capacity; 10.68 GiB already allocated; 95.31 MiB free; 10.77 GiB reserved in total by PyTorch)
`
Please how do I resolve this


Fix marian slow test

Fix slow failing test that depended on old seq2seq_batch logic.


FAILED tests/test_modeling_marian.py::TestMarian_EN_DE_More::test_forward


Logging doc

Adds documentation for the new centralized logger.


Distill marian

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #{issue number}


move wandb/comet logger init to train() to allow parallel logging

Moving the logger setup to the `train()` function allows parallel runs (e.g. in hyperparameter search) to log each run individually.

Alternative to #6791


Printing probabilities

Hi,

I apologize if that's a stupid question. How can I print probabilities during inference produced by the softmax layer with BertForSequenceClassification?


unexpected behavior on RoBERTa tokenizer when using additional special tokens

## Environment info
- `transformers` version: 3.0.2
- Platform: Windows-10-10.0.18362-SP0
- Python version: 3.8.3
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: <no
- Using distributed or parallel set-up in script?: no

### Who can help
 tokenizers: @mfuntowicz

## Information

When trying to tokenize using RoBERTa tokenized and a special token and using add_prefix_space=True, the token following the special token does not get a space.

## To reproduce

Steps to reproduce the behavior:

1. run the following code

```
tokenizer = RobertaTokenizer.from_pretrained("roberta-base")
tokenizer.add_special_tokens('[d_s]')
print(tokenizer('[d_s] test', add_prefix_space=True))
print((tokenizer('test', add_prefix_space=True))
```

## output
{'input_ids': [0, 1296, 2], 'attention_mask': [1, 1, 1]}
{'input_ids': [0, 50271, 21959, 2], 'attention_mask': [1, 1, 1, 1]}

## Expected behavior
{'input_ids': [0, 1296, 2], 'attention_mask': [1, 1, 1]}
{'input_ids': [0, 50271, 1296, 2], 'attention_mask': [1, 1, 1, 1]}

The second token should not change because of the first one.


Fix resuming training for Windows

Fixes #6720


Separate implementation for Torch-Scriptable BERT model

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #5067

Separate (re-)implementation of BERT such that it can be used with TorchScript's script() method (not just trace). This allows for better/more reliable handling of internal model logic and removes the requirement for fixed input size, resulting in large speedups when average input size is << than max input size.

Tests duplicate all the ones on the original BERT model, with the only fundamental difference being the test_torchscript* tests, which now use script() rather than trace().


Fix in Adafactor docstrings


Pegasus: replication and distillation results

Help wanted

Extra attention is needed, help appreciated

### Replication
[link](https://github.com/google-research/pegasus)

mixed & stochastic column of this [table](https://github.com/google-research/pegasus#results-update)

| dataset | Authors| This Repo| best bart | best bart name
| ---- | ----|----|----|----|
| xsum | 47.60/24.83/39.64|  46.87/24.46/39.15|22.32/37.39|distilbart-xsum-12-6|
| cnn_dailymail | 44.16/21.56/41.30| see comment|21.26/30.59|distilbart-cnn-12-6|
| newsroom | 45.07/33.39/41.28 |41.03/29.83/36.96|
| multi_news | 47.65/18.75/24.95|47.58/19.0/24.77|
| gigaword | 39.65/20.47/36.76|39.79/20.56/36.80|
| wikihow | 46.39/22.12/38.41 *|46.85/23.64/28.73|
| reddit_tifu | 27.99/9.81/22.94|32.75/11.68/24.97|
| big_patent |52.29/33.08/41.66 *|
| arxiv | 44.21/16.95/25.67|44.83/17.34/25.60|
| pubmed | 45.97/20.15/28.25|45.40/19.42/26.93|
| aeslc | 37.68/21.25/36.51|37.09/21.40/35.93|
| billsum | 59.67/41.58/47.59|56.18/39.94/45.39|

+ (* (authors footnote)) the numbers of wikihow and big_patent datasets are not comparable because of change in tokenization and data

#### Final Update (2020-10-16)
Mission accomplished thanks to the work of @patil-suraj, and @stas00 !

The above table now shows that our results are close enough.
We suspect differences are due to treatment of the `<n>` character that pegasus generates and slightly different beam search implementations.

[Link to Spreadsheet with timing data](https://docs.google.com/spreadsheets/d/1ODfoK-tXOV6TLXDMnujdGLtFhA8oVTy-Cv6Ib6qKgWk/edit?usp=sharing)

Questions about specific results should be asked on the forums/separate issues with @stas00, @patil-suraj, and @sshleifer tagged.


Adding another translation example

- IWSLT 2017 (should be added to `nlp` via
https://github.com/huggingface/nlp/pull/470#issue-462074344. (Currently
coded file here)
- NL-EN, should hopefully work for any language pair (Could be checked).
- Training loop in lightning, poor training logging, notably no
translation example which are probably important.


Update `convert_bart` script to allow loading nonstandard model architectures and custom pretrained Fairseq models.

**edit 30/10/20** - this approach doesn't work anymore given the differences in interface from when I submitted it it now (mostly because of Hydra configs in fairseq). Closed.

# 🚀 Feature request

Allow the `convert_bart` script to load custom BART pre-trained models from fairseq.

## Motivation

The current BART conversion script to migrate from Fairseq to Huggingface assumes that you are using predefined  shape/parameter values from a BART model. There might be a usecase where you want to experiment with your own Seq2Seq pretraining and train a smaller BART model (i.e. cutting dimensions by 1/3rd) because of small datasets/pruning/some other reason. There presently isn't a dedicated Fairseq tutorial on training your own BART models but it is possible using the "denoising" task and the appropriate parameters (e.g. [preprocess](https://gist.github.com/tomsherborne/ab1a5a28f9d843cf633d6f7843e96a63) and [train](https://gist.github.com/tomsherborne/ae3529375b7a538a1b03a53f34850234)). This is what I have been trying with a small dataset and a smaller BART model.

Then if you want to use this model within `transformers` - the conversion script doesn't work because (a) `load_xsum_checkpoint` assumes your model should be the shape of `bart.large.cnn` and rejects the incorrect tensor sizes and (b) an additional weight `decoder.output_projection.weight` needs to be ignored.

To convert correctly to `transformers` and pass all the tests - I found you can switch from `torch.hub.load` to `fairseq.models.bart.BARTModel.from_pretrained` which calls `torch.hub` internally and gives you the same output model. This means you can convert local, custom BART models into `transformers` models and use them in downstream tasks or upload them to the models archive.

## Your contribution

I made my own version of `convert_bart` [here](https://gist.github.com/tomsherborne/e7b629ee9cf0618febb211683a410ce5) which passes the scripts own tests. This could be improved/refactored if this feature is still useful. There could be a better way to do all this that I have missed. There's a hack because I call `torch.load` and `from_pretrained()` on the same weights and I'm not sure if they are both needed.

The execution is bit ugly because I set `hf_config` as a path to a JSON file created from `BartConfig.from_pretrained()` and then manually adjusting to the new model size.
```
SCRIPT_PATH="${HOME}/ed/sp/transformers/src/transformers/convert_bart_original_pytorch_checkpoint_to_pytorch.py"
TEMPLATE_ARCHIVE=""/Users/tom/ed/sp/pretrain/runs/fairseq/bart_base_enqa_small/bart-mini.tar.gz""
CHECKPOINT_PATH="${HOME}/ed/sp/pretrain/runs/fairseq/bart_base_enqa_small/ckpt/checkpoint_best.pt"
DUMP_PATH="${HOME}/ed/sp/pretrain/runs/hf_fromfairseq/bart-mini"
CONFIG_PATH="${HOME}/ed/sp/pretrain/config/hf_bart_configs/bart-mini/config.json"

source ~/miniconda3/bin/activate $ENV_NAME

python $SCRIPT_PATH --hf_config $CONFIG_PATH --model_template $TEMPLATE_ARCHIVE $CHECKPOINT_PATH $DUMP_PATH

```


TF Flaubert w/ pre-norm

#5614 fixed TF Flaubert without pre-norm, but didn't fix it with the pre-norm.

Fixes #6084


Model cards for loodos

model card

Related to pretrained model cards

Adds cards for 6 models in the ./models/loodos directory.

albert-base-turkish-uncased
bert-base-turkish-uncased
electra-base-turkish-uncased
electra-base-turkish-64k-uncased-discriminator
electra-small-turkish-cased-discriminator
electra-small-turkish-uncased-discriminator


pegasus-large: Can we have input text descriptions more than the maximum input length of 512?

As mentioned pegasus paper, authors used sinusoidal positional encoding to make sure the model can work with inputs longer than default 512?

So with huggingface implementation can we use inputs with longer lengths?


fix typo in comments (modeling_bert)


tokenization_gpt2 save vocabulary is not saving special tokens

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 2.8.0
- Platform:  linux
- Python version: 3.6.10
- PyTorch version (GPU?): GPU
- Tensorflow version (GPU?): -
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, GPT2, XLM: @LysandreJik
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->

## Information

Model I am using (Bert, XLNet ...):
GPT2

The problem arises when using:
 the official example scripts: (give details below)

The tasks I am working on is:

my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Load a pre-trained tokenizer
2. Add your intended special tokens: `tokenizer.add_tokens(SPECIAL_TOKENS_LIST)`
3. Save your tokenizer's vocabulary with: `tokenizer.save_vocabulary(PATH)`

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior
With the current `save_vocabulary` function, we are just saving the predefined tokens:

`with open(vocab_file, "w", encoding="utf-8") as f:
   f.write(json.dumps(self.encoder, ensure_ascii=False))`

This line should be modified as follow to save the special tokens as well:

`vocab_with_special_tokens = dict(self.encoder, **self.added_tokens_encoder)
with open(vocab_file, "w", encoding="utf-8") as f:
            f.write(json.dumps(vocab_with_special_tokens, ensure_ascii=False))`

<!-- A clear and concise description of what you would expect to happen. -->


RAM MemoryError

# ❓ Questions & Help

I was wondering is the RAM MemoryError issue solved?
I encountered this issue because my 128 GB RAM memory could not load all 48 GB data.
There are some discussions before, such as #6636, #3388, #1290, #4009
However, I don't see there is lazyDataLoader right now.
Could you provide any hints about how to deal with a large dataset?
Thanks!


DistributedSortishSampler

Examples

Which is related to examples in general

In examples/seq2seq/finetune.py,
`--sortish_sampler  --gpus 2` will raise an assertion error, but if you remove the assert, it will raise another error.   Ideally we should make a Seq2SeqDataset.get_distributed_sortish_sampler method and use it in the relevant case.


[s2s] distill: --normalize_hidden --supervise_forward

This appears to help pegasus and marian distillation.

running bart-large-xsum-12-3 baseline.

- No impact on bart-large-xsum distillation.
- +1 BLEU for Marian
- +20 ROUGE for pegasus (impossible to do anything without.)
- verified that the `torch.stack` math is identical to the old for loop math.


[s2s] command line args for faster val steps

cc @patil-suraj


Model.fit on GPT2 and TPUs

## Environment info
- `transformers` version: 3.0.2
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (False)
- Tensorflow version (GPU?): 2.3.0 (False)
- Using GPU in script?:  No
- Using distributed or parallel set-up in script?: Yes

## Information

Model I am using: GPT2

@jplu When using the keras model.fit() method, it looks like there's a problem with logits and tensors: `Compilation failure: logits and labels must have the same first dimension`. Setting `from_logits = False` doesn't seem to resolve the problem. Any suggestion on how to change model compilation or the dataset to fix this?

```
# TPU and Strategy Initialization
resolver = tf.distribute.cluster_resolver.TPUClusterResolver(tpu='grpc://' + os.environ['COLAB_TPU_ADDR'])
tf.config.experimental_connect_to_cluster(resolver)
tf.tpu.experimental.initialize_tpu_system(resolver)
strategy = tf.distribute.TPUStrategy(resolver)

# Load and compile model
with strategy.scope():
    model = TFGPT2LMHeadModel.from_pretrained('gpt2-medium')
    model.resize_token_embeddings(len(tokenizer))
    optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5)
    loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
    model.compile(optimizer = optimizer,
              loss = loss)

# Format inputs
input_ids = tf.convert_to_tensor(input_ids)
attention_mask = tf.convert_to_tensor(attention_mask)
labels = tf.convert_to_tensor(labels)

# Create dataset
dataset = tf.data.Dataset.from_tensor_slices(({'input_ids': input_ids, 'attention_mask': attention_mask}, labels))
num_examples = tf.data.experimental.cardinality(dataset).numpy()
train_dataset = dataset.repeat().shuffle(num_examples).batch(8)

# Train model
model.fit(x = train_dataset, epochs = 1, batch_size = 8, steps_per_epoch = num_examples)
---------------------------------------------------------------------------
InvalidArgumentError                      Traceback (most recent call last)
<ipython-input-10-4efb969ef56e> in <module>()
---> 23 model.fit(x = train_dataset, epochs = 1, batch_size = 8, steps_per_epoch = num_examples)

InvalidArgumentError: 9 root error(s) found.
  (0) Invalid argument: {{function_node __inference_train_function_121591}} Compilation failure: logits and labels must have the same first dimension, got logits shape [32768,64] and labels shape [1024]
	 [[{{node sparse_categorical_crossentropy_24/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits}}]]
	TPU compilation failed
	 [[tpu_compile_succeeded_assert/_142128106034484350/_6]]
	 [[tpu_compile_succeeded_assert/_142128106034484350/_6/_223]]
  (1) Invalid argument: {{function_node __inference_train_function_121591}} Compilation failure: logits and labels must have the same first dimension, got logits shape [32768,64] and labels shape [1024]
	 [[{{node sparse_categorical_crossentropy_24/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits}}]]
	TPU compilation failed
	 [[tpu_compile_succeeded_assert/_142128106034484350/_6]]
	 [[tpu_compile_succeeded_assert/_142128106034484350/_6/_307]]
```


Update ONNX notebook to include section on quantization.

Added section regarding quantization with performance comparison against PyTorch on CPU.

Signed-off-by: Morgan Funtowicz <morgan@huggingface.co>


Related to abstractive text summarization

I was wondering if we could use XLM or XLM-R for abstractive text summarization.


No attribute '_mp_fn' when fine-tuning mbart for en-ro translation task using TPU

I followed the TPU example in the [examples folder](9https://github.com/huggingface/transformers/tree/master/examples) and found xla_spawn.py calls
`xmp.spawn(mod._mp_fn, args=(), nprocs=args.num_cores)`
and [fine-tune.py](https://github.com/huggingface/transformers/blob/master/examples/seq2seq/finetune.py) does not have the "_mp_fn" found in some training scripts.
I get
```
Traceback (most recent call last):
  File "examples/xla_spawn.py", line 72, in <module>
    main()
  File "examples/xla_spawn.py", line 68, in main
    xmp.spawn(mod._mp_fn, args=(), nprocs=args.num_cores)
AttributeError: module 'finetune' has no attribute '_mp_fn'
```
Tried to fix it by adding the _mp_fn:
```
#def _mp_fn(index):
    # For xla_spawn (TPUs)
#    pass
    #main()
```
with and without args `main(args)` but neither worked.


regarding the max token length of longformer

In the encode_plus function  of the Tokenizer , there is a argument called max_length whose default value is 4096.
So is it possible to increase the max token length  beyond 4096 or the maximum value of max_length argument is 4096
Can anyone clear my doubt

Thanks in advance !!!!


Add model card for singbert lite.

model card

Related to pretrained model cards

Add model card for singbert lite and update widget for singbert and singbert large.


Loading a converted pytorch model in huggingface transformers properly

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
I converted a pre-trained tf model to pytorch using the following function.
```
def convert_tf_checkpoint_to_pytorch(*, tf_checkpoint_path, albert_config_file, pytorch_dump_path):
    # Initialise PyTorch model
    config = AlbertConfig.from_json_file(albert_config_file)
    print("Building PyTorch model from configuration: {}".format(str(config)))
    model = AlbertForPreTraining(config)

    # Load weights from tf checkpoint
    load_tf_weights_in_albert(model, config, tf_checkpoint_path)

    # Save pytorch-model
    print("Save PyTorch model to {}".format(pytorch_dump_path))
    torch.save(model.state_dict(), pytorch_dump_path)
```
I am loading the converted model and encoding sentences in the following way:
```
def vectorize_sentence(text):
    albert_tokenizer = AlbertTokenizer.from_pretrained("albert-base-v2")
    config = AlbertConfig.from_pretrained(config_path, output_hidden_states=True)
    model = TFAlbertModel.from_pretrained(pytorch_dir, config=config, from_pt=True)
    e = albert_tokenizer.encode(text, max_length=512)
    model_input = tf.constant(e)[None, :]  # Batch size 1
    output = model(model_input)

    v = [0] * 768
    # generate sentence vectors by averaging the word vectors
    for i in range(1, len(model_input[0]) - 1):
        v = v + output[0][0][i].numpy()

    vector = v/len(model_input[0])
    return vector
```
However while loading the model, a warning comes up:

> Some weights or buffers of the PyTorch model TFAlbertModel were not
> initialized from the TF 2.0 model and are newly initialized:
> ['predictions.LayerNorm.bias', 'predictions.dense.weight',
> 'predictions.LayerNorm.weight', 'sop_classifier.classifier.bias',
> 'predictions.dense.bias', 'sop_classifier.classifier.weight',
> 'predictions.decoder.bias', 'predictions.bias',
> 'predictions.decoder.weight'] You should probably TRAIN this model on
> a down-stream task to be able to use it for predictions and inference.

Can anyone tell me if I am doing anything wrong? What does the warning mean? I saw issue #5588. Don't know if my issue is the same as this.

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:
https://stackoverflow.com/questions/63648380/loading-a-converted-pytorch-model-in-huggingface-transformers-properly


Fixed open in colab link

Changed non existant link - https://colab.research.google.com/github/huggingface/transformers/blob/master/notebooks/03-pipelines.ipynb
to - https://colab.research.google.com/github/huggingface/transformers/blob/master/notebooks/03-pipelines.ipynb


How to convert '.bin' model to '.onnx'

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->

<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**:


How to use encode_plus to force padding to specific length

I am using the following code in the __getitem__() method of my dataset:

`

    class MyDataset(Dataset):
        def __init__(self, myargs):

            #other code here
            self.tokenizer = BertTokenizer.from_pretrained('bert-base-cased)
            self.max_len =50

`

__getitem__ method:

`

        def __getitem__(self, idx):
           #other code here
           text_encoded = self.tokenizer.encode_plus(
              text,
              add_special_tokens=True,
              padding=True,
              truncation=True,
              max_length=self.max_len,
              return_token_type_ids=False,
              pad_to_max_length=True,
              return_attention_mask=True,
              return_tensors='pt',)

        input_ids=text_encoded['input_ids'].flatten()
        attention_mask=text_encoded['attention_mask'].flatten()

    >>input_ids: torch.Size([9]), attention_mask: torch.Size([9])
    >>input_ids: torch.Size([21]), attention_mask: torch.Size([21])

`

Even though I have set padding and truncation to True and set a max_length the returned lengths of the input_ids and attention_mask values in the retured text_encoded dict are variable depending on the input text. Is this normal behaviour? if so how can I ensure that ecery rturned sample is padded out to and truncated at a specific length?


[s2s README] link to cnn dataset with empty lines removed

<!-- This line specifies which issue to close after the pull request is merged. -->
Fixes #{issue number}


How to generate on multiple GPUs?

Hi!

How would I run generation on multiple GPUs at the same time? Running model.generate on a DataParallel layer isn't possible, and model.module.generate run on a single GPU.

Any advice would be appreciated!


Bert transformer issue

# ❓ Questions & Help

<!-- The GitHub issue tracker is primarly intended for bugs, feature requests,
     new models and benchmarks, and migration questions. For all other questions,
     we direct you to the Hugging Face forum: https://discuss.huggingface.co/ .
     You can also try Stack Overflow (SO) where a whole community of PyTorch and
     Tensorflow enthusiast can help you out. In this case, make sure to tag your
     question with the right deep learning framework as well as the
     huggingface-transformers tag:
     https://stackoverflow.com/questions/tagged/huggingface-transformers
     -->

## Details
<!-- Description of your issue -->
Token indices sequence length is longer than the specified maximum sequence length for this model (5 > 512). Running this sequence through the model will result in indexing errors
<!-- You should first ask your question on the forum or SO, and only if
     you didn't get an answer ask it here on GitHub. -->
**A link to original question on the forum/Stack Overflow**: