Problems with using LongFormer

I am according to the official longformer lot (https://github.com/allenai/longformer) provides methods to use, I use in the code of parts as follows:
`

            tokenizer_class = BertTokenizer
            model_class = LongformerModel
            # directory is fine
            pretrained_weights = self.pretrainedBertPath
            tokenizer = tokenizer_class.from_pretrained(pretrained_weights)
            model = model_class.from_pretrained('longformer-base-4096', gradient_checkpointing=True)
            # add_special_tokens will add start and end token
            input_ids = torch.tensor([tokenizer.encode(text, add_special_tokens=False)])

`

This warning appeared:
`

    Some weights of the model checkpoint at longformer-base-4096 were not used when initializing LongformerModel: ['roberta.embeddings.word_embeddings.weight', 'roberta.embeddings.position_embeddings.weight', 'roberta.embeddings.token_type_embeddings.weight', 'roberta.embeddings.LayerNorm.weight', 'roberta.embeddings.LayerNorm.bias', 'roberta.encoder.layer.0.attention.self.query.weight', 'roberta.encoder.layer.0.attention.self.query.bias', 'roberta.encoder.layer.0.attention.self.key.weight', 'roberta.encoder.layer.0.attention.self.key.bias', 'roberta.encoder.layer.0.attention.self.value.weight', 'roberta.encoder.layer.0.attention.self.value.bias', 'roberta.encoder.layer.0.attention.self.query_global.weight', 'roberta.encoder.layer.0.attention.self.query_global.bias', 'roberta.encoder.layer.0.attention.self.key_global.weight', 'roberta.encoder.layer.0.attention.self.key_global.bias', 'roberta.encoder.layer.0.attention.self.value_global.weight', 'roberta.encoder.layer.0.attention.self.value_global.bias', 'roberta.encoder.layer.0.attention.output.dense.weight', 'roberta.encoder.layer.0.attention.output.dense.bias', 'roberta.encoder.layer.0.attention.output.LayerNorm.weight', 'roberta.encoder.layer.0.attention.output.LayerNorm.bias', 'roberta.encoder.layer.0.intermediate.dense.weight', 'roberta.encoder.layer.0.intermediate.dense.bias', 'roberta.encoder.layer.0.output.dense.weight', 'roberta.encoder.layer.0.output.dense.bias', 'roberta.encoder.layer.0.output.LayerNorm.weight', 'roberta.encoder.layer.0.output.LayerNorm.bias', 'roberta.encoder.layer.1.attention.self.query.weight', 'roberta.encoder.layer.1.attention.self.query.bias', 'roberta.encoder.layer.1.attention.self.key.weight', 'roberta.encoder.layer.1.attention.self.key.bias', 'roberta.encoder.layer.1.attention.self.value.weight', 'roberta.encoder.layer.1.attention.self.value.bias', 'roberta.encoder.layer.1.attention.self.query_global.weight', 'roberta.encoder.layer.1.attention.self.query_global.bias', 'roberta.encoder.layer.1.attention.self.key_global.weight', 'roberta.encoder.layer.1.attention.self.key_global.bias', 'roberta.encoder.layer.1.attention.self.value_global.weight', 'roberta.encoder.layer.1.attention.self.value_global.bias', 'roberta.encoder.layer.1.attention.output.dense.weight', 'roberta.encoder.layer.1.attention.output.dense.bias', 'roberta.encoder.layer.1.attention.output.LayerNorm.weight', 'roberta.encoder.layer.1.attention.output.LayerNorm.bias', 'roberta.encoder.layer.1.intermediate.dense.weight', 'roberta.encoder.layer.1.intermediate.dense.bias', 'roberta.encoder.layer.1.output.dense.weight', 'roberta.encoder.layer.1.output.dense.bias', 'roberta.encoder.layer.1.output.LayerNorm.weight', 'roberta.encoder.layer.1.output.LayerNorm.bias', 'roberta.encoder.layer.2.attention.self.query.weight', 'roberta.encoder.layer.2.attention.self.query.bias', 'roberta.encoder.layer.2.attention.self.key.weight', 'roberta.encoder.layer.2.attention.self.key.bias', 'roberta.encoder.layer.2.attention.self.value.weight', 'roberta.encoder.layer.2.attention.self.value.bias', 'roberta.encoder.layer.2.attention.self.query_global.weight', 'roberta.encoder.layer.2.attention.self.query_global.bias', 'roberta.encoder.layer.2.attention.self.key_global.weight', 'roberta.encoder.layer.2.attention.self.key_global.bias', 'roberta.encoder.layer.2.attention.self.value_global.weight', 'roberta.encoder.layer.2.attention.self.value_global.bias', 'roberta.encoder.layer.2.attention.output.dense.weight', 'roberta.encoder.layer.2.attention.output.dense.bias', 'roberta.encoder.layer.2.attention.output.LayerNorm.weight', 'roberta.encoder.layer.2.attention.output.LayerNorm.bias', 'roberta.encoder.layer.2.intermediate.dense.weight', 'roberta.encoder.layer.2.intermediate.dense.bias', 'roberta.encoder.layer.2.output.dense.weight', 'roberta.encoder.layer.2.output.dense.bias', 'roberta.encoder.layer.2.output.LayerNorm.weight', 'roberta.encoder.layer.2.output.LayerNorm.bias', 'roberta.encoder.layer.3.attention.self.query.weight', 'roberta.encoder.layer.3.attention.self.query.bias', 'roberta.encoder.layer.3.attention.self.key.weight', 'roberta.encoder.layer.3.attention.self.key.bias', 'roberta.encoder.layer.3.attention.self.value.weight', 'roberta.encoder.layer.3.attention.self.value.bias', 'roberta.encoder.layer.3.attention.self.query_global.weight', 'roberta.encoder.layer.3.attention.self.query_global.bias', 'roberta.encoder.layer.3.attention.self.key_global.weight', 'roberta.encoder.layer.3.attention.self.key_global.bias', 'roberta.encoder.layer.3.attention.self.value_global.weight', 'roberta.encoder.layer.3.attention.self.value_global.bias', 'roberta.encoder.layer.3.attention.output.dense.weight', 'roberta.encoder.layer.3.attention.output.dense.bias', 'roberta.encoder.layer.3.attention.output.LayerNorm.weight', 'roberta.encoder.layer.3.attention.output.LayerNorm.bias', 'roberta.encoder.layer.3.intermediate.dense.weight', 'roberta.encoder.layer.3.intermediate.dense.bias', 'roberta.encoder.layer.3.output.dense.weight', 'roberta.encoder.layer.3.output.dense.bias', 'roberta.encoder.layer.3.output.LayerNorm.weight', 'roberta.encoder.layer.3.output.LayerNorm.bias', 'roberta.encoder.layer.4.attention.self.query.weight', 'roberta.encoder.layer.4.attention.self.query.bias', 'roberta.encoder.layer.4.attention.self.key.weight', 'roberta.encoder.layer.4.attention.self.key.bias', 'roberta.encoder.layer.4.attention.self.value.weight', 'roberta.encoder.layer.4.attention.self.value.bias', 'roberta.encoder.layer.4.attention.self.query_global.weight', 'roberta.encoder.layer.4.attention.self.query_global.bias', 'roberta.encoder.layer.4.attention.self.key_global.weight', 'roberta.encoder.layer.4.attention.self.key_global.bias', 'roberta.encoder.layer.4.attention.self.value_global.weight', 'roberta.encoder.layer.4.attention.self.value_global.bias', 'roberta.encoder.layer.4.attention.output.dense.weight', 'roberta.encoder.layer.4.attention.output.dense.bias', 'roberta.encoder.layer.4.attention.output.LayerNorm.weight', 'roberta.encoder.layer.4.attention.output.LayerNorm.bias', 'roberta.encoder.layer.4.intermediate.dense.weight', 'roberta.encoder.layer.4.intermediate.dense.bias', 'roberta.encoder.layer.4.output.dense.weight', 'roberta.encoder.layer.4.output.dense.bias', 'roberta.encoder.layer.4.output.LayerNorm.weight', 'roberta.encoder.layer.4.output.LayerNorm.bias', 'roberta.encoder.layer.5.attention.self.query.weight', 'roberta.encoder.layer.5.attention.self.query.bias', 'roberta.encoder.layer.5.attention.self.key.weight', 'roberta.encoder.layer.5.attention.self.key.bias', 'roberta.encoder.layer.5.attention.self.value.weight', 'roberta.encoder.layer.5.attention.self.value.bias', 'roberta.encoder.layer.5.attention.self.query_global.weight', 'roberta.encoder.layer.5.attention.self.query_global.bias', 'roberta.encoder.layer.5.attention.self.key_global.weight', 'roberta.encoder.layer.5.attention.self.key_global.bias', 'roberta.encoder.layer.5.attention.self.value_global.weight', 'roberta.encoder.layer.5.attention.self.value_global.bias', 'roberta.encoder.layer.5.attention.output.dense.weight', 'roberta.encoder.layer.5.attention.output.dense.bias', 'roberta.encoder.layer.5.attention.output.LayerNorm.weight', 'roberta.encoder.layer.5.attention.output.LayerNorm.bias', 'roberta.encoder.layer.5.intermediate.dense.weight', 'roberta.encoder.layer.5.intermediate.dense.bias', 'roberta.encoder.layer.5.output.dense.weight', 'roberta.encoder.layer.5.output.dense.bias', 'roberta.encoder.layer.5.output.LayerNorm.weight', 'roberta.encoder.layer.5.output.LayerNorm.bias', 'roberta.encoder.layer.6.attention.self.query.weight', 'roberta.encoder.layer.6.attention.self.query.bias', 'roberta.encoder.layer.6.attention.self.key.weight', 'roberta.encoder.layer.6.attention.self.key.bias', 'roberta.encoder.layer.6.attention.self.value.weight', 'roberta.encoder.layer.6.attention.self.value.bias', 'roberta.encoder.layer.6.attention.self.query_global.weight', 'roberta.encoder.layer.6.attention.self.query_global.bias', 'roberta.encoder.layer.6.attention.self.key_global.weight', 'roberta.encoder.layer.6.attention.self.key_global.bias', 'roberta.encoder.layer.6.attention.self.value_global.weight', 'roberta.encoder.layer.6.attention.self.value_global.bias', 'roberta.encoder.layer.6.attention.output.dense.weight', 'roberta.encoder.layer.6.attention.output.dense.bias', 'roberta.encoder.layer.6.attention.output.LayerNorm.weight', 'roberta.encoder.layer.6.attention.output.LayerNorm.bias', 'roberta.encoder.layer.6.intermediate.dense.weight', 'roberta.encoder.layer.6.intermediate.dense.bias', 'roberta.encoder.layer.6.output.dense.weight', 'roberta.encoder.layer.6.output.dense.bias', 'roberta.encoder.layer.6.output.LayerNorm.weight', 'roberta.encoder.layer.6.output.LayerNorm.bias', 'roberta.encoder.layer.7.attention.self.query.weight', 'roberta.encoder.layer.7.attention.self.query.bias', 'roberta.encoder.layer.7.attention.self.key.weight', 'roberta.encoder.layer.7.attention.self.key.bias', 'roberta.encoder.layer.7.attention.self.value.weight', 'roberta.encoder.layer.7.attention.self.value.bias', 'roberta.encoder.layer.7.attention.self.query_global.weight', 'roberta.encoder.layer.7.attention.self.query_global.bias', 'roberta.encoder.layer.7.attention.self.key_global.weight', 'roberta.encoder.layer.7.attention.self.key_global.bias', 'roberta.encoder.layer.7.attention.self.value_global.weight', 'roberta.encoder.layer.7.attention.self.value_global.bias', 'roberta.encoder.layer.7.attention.output.dense.weight', 'roberta.encoder.layer.7.attention.output.dense.bias', 'roberta.encoder.layer.7.attention.output.LayerNorm.weight', 'roberta.encoder.layer.7.attention.output.LayerNorm.bias', 'roberta.encoder.layer.7.intermediate.dense.weight', 'roberta.encoder.layer.7.intermediate.dense.bias', 'roberta.encoder.layer.7.output.dense.weight', 'roberta.encoder.layer.7.output.dense.bias', 'roberta.encoder.layer.7.output.LayerNorm.weight', 'roberta.encoder.layer.7.output.LayerNorm.bias', 'roberta.encoder.layer.8.attention.self.query.weight', 'roberta.encoder.layer.8.attention.self.query.bias', 'roberta.encoder.layer.8.attention.self.key.weight', 'roberta.encoder.layer.8.attention.self.key.bias', 'roberta.encoder.layer.8.attention.self.value.weight', 'roberta.encoder.layer.8.attention.self.value.bias', 'roberta.encoder.layer.8.attention.self.query_global.weight', 'roberta.encoder.layer.8.attention.self.query_global.bias', 'roberta.encoder.layer.8.attention.self.key_global.weight', 'roberta.encoder.layer.8.attention.self.key_global.bias', 'roberta.encoder.layer.8.attention.self.value_global.weight', 'roberta.encoder.layer.8.attention.self.value_global.bias', 'roberta.encoder.layer.8.attention.output.dense.weight', 'roberta.encoder.layer.8.attention.output.dense.bias', 'roberta.encoder.layer.8.attention.output.LayerNorm.weight', 'roberta.encoder.layer.8.attention.output.LayerNorm.bias', 'roberta.encoder.layer.8.intermediate.dense.weight', 'roberta.encoder.layer.8.intermediate.dense.bias', 'roberta.encoder.layer.8.output.dense.weight', 'roberta.encoder.layer.8.output.dense.bias', 'roberta.encoder.layer.8.output.LayerNorm.weight', 'roberta.encoder.layer.8.output.LayerNorm.bias', 'roberta.encoder.layer.9.attention.self.query.weight', 'roberta.encoder.layer.9.attention.self.query.bias', 'roberta.encoder.layer.9.attention.self.key.weight', 'roberta.encoder.layer.9.attention.self.key.bias', 'roberta.encoder.layer.9.attention.self.value.weight', 'roberta.encoder.layer.9.attention.self.value.bias', 'roberta.encoder.layer.9.attention.self.query_global.weight', 'roberta.encoder.layer.9.attention.self.query_global.bias', 'roberta.encoder.layer.9.attention.self.key_global.weight', 'roberta.encoder.layer.9.attention.self.key_global.bias', 'roberta.encoder.layer.9.attention.self.value_global.weight', 'roberta.encoder.layer.9.attention.self.value_global.bias', 'roberta.encoder.layer.9.attention.output.dense.weight', 'roberta.encoder.layer.9.attention.output.dense.bias', 'roberta.encoder.layer.9.attention.output.LayerNorm.weight', 'roberta.encoder.layer.9.attention.output.LayerNorm.bias', 'roberta.encoder.layer.9.intermediate.dense.weight', 'roberta.encoder.layer.9.intermediate.dense.bias', 'roberta.encoder.layer.9.output.dense.weight', 'roberta.encoder.layer.9.output.dense.bias', 'roberta.encoder.layer.9.output.LayerNorm.weight', 'roberta.encoder.layer.9.output.LayerNorm.bias', 'roberta.encoder.layer.10.attention.self.query.weight', 'roberta.encoder.layer.10.attention.self.query.bias', 'roberta.encoder.layer.10.attention.self.key.weight', 'roberta.encoder.layer.10.attention.self.key.bias', 'roberta.encoder.layer.10.attention.self.value.weight', 'roberta.encoder.layer.10.attention.self.value.bias', 'roberta.encoder.layer.10.attention.self.query_global.weight', 'roberta.encoder.layer.10.attention.self.query_global.bias', 'roberta.encoder.layer.10.attention.self.key_global.weight', 'roberta.encoder.layer.10.attention.self.key_global.bias', 'roberta.encoder.layer.10.attention.self.value_global.weight', 'roberta.encoder.layer.10.attention.self.value_global.bias', 'roberta.encoder.layer.10.attention.output.dense.weight', 'roberta.encoder.layer.10.attention.output.dense.bias', 'roberta.encoder.layer.10.attention.output.LayerNorm.weight', 'roberta.encoder.layer.10.attention.output.LayerNorm.bias', 'roberta.encoder.layer.10.intermediate.dense.weight', 'roberta.encoder.layer.10.intermediate.dense.bias', 'roberta.encoder.layer.10.output.dense.weight', 'roberta.encoder.layer.10.output.dense.bias', 'roberta.encoder.layer.10.output.LayerNorm.weight', 'roberta.encoder.layer.10.output.LayerNorm.bias', 'roberta.encoder.layer.11.attention.self.query.weight', 'roberta.encoder.layer.11.attention.self.query.bias', 'roberta.encoder.layer.11.attention.self.key.weight', 'roberta.encoder.layer.11.attention.self.key.bias', 'roberta.encoder.layer.11.attention.self.value.weight', 'roberta.encoder.layer.11.attention.self.value.bias', 'roberta.encoder.layer.11.attention.self.query_global.weight', 'roberta.encoder.layer.11.attention.self.query_global.bias', 'roberta.encoder.layer.11.attention.self.key_global.weight', 'roberta.encoder.layer.11.attention.self.key_global.bias', 'roberta.encoder.layer.11.attention.self.value_global.weight', 'roberta.encoder.layer.11.attention.self.value_global.bias', 'roberta.encoder.layer.11.attention.output.dense.weight', 'roberta.encoder.layer.11.attention.output.dense.bias', 'roberta.encoder.layer.11.attention.output.LayerNorm.weight', 'roberta.encoder.layer.11.attention.output.LayerNorm.bias', 'roberta.encoder.layer.11.intermediate.dense.weight', 'roberta.encoder.layer.11.intermediate.dense.bias', 'roberta.encoder.layer.11.output.dense.weight', 'roberta.encoder.layer.11.output.dense.bias', 'roberta.encoder.layer.11.output.LayerNorm.weight', 'roberta.encoder.layer.11.output.LayerNorm.bias', 'roberta.pooler.dense.weight', 'roberta.pooler.dense.bias', 'lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight']
    - This IS expected if you are initializing LongformerModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
    - This IS NOT expected if you are initializing LongformerModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
    Some weights of LongformerModel were not initialized from the model checkpoint at longformer-base-4096 and are newly initialized: ['embeddings.word_embeddings.weight', 'embeddings.position_embeddings.weight', 'embeddings.token_type_embeddings.weight', 'embeddings.LayerNorm.weight', 'embeddings.LayerNorm.bias', 'encoder.layer.0.attention.self.query.weight', 'encoder.layer.0.attention.self.query.bias', 'encoder.layer.0.attention.self.key.weight', 'encoder.layer.0.attention.self.key.bias', 'encoder.layer.0.attention.self.value.weight', 'encoder.layer.0.attention.self.value.bias', 'encoder.layer.0.attention.self.query_global.weight', 'encoder.layer.0.attention.self.query_global.bias', 'encoder.layer.0.attention.self.key_global.weight', 'encoder.layer.0.attention.self.key_global.bias', 'encoder.layer.0.attention.self.value_global.weight', 'encoder.layer.0.attention.self.value_global.bias', 'encoder.layer.0.attention.output.dense.weight', 'encoder.layer.0.attention.output.dense.bias', 'encoder.layer.0.attention.output.LayerNorm.weight', 'encoder.layer.0.attention.output.LayerNorm.bias', 'encoder.layer.0.intermediate.dense.weight', 'encoder.layer.0.intermediate.dense.bias', 'encoder.layer.0.output.dense.weight', 'encoder.layer.0.output.dense.bias', 'encoder.layer.0.output.LayerNorm.weight', 'encoder.layer.0.output.LayerNorm.bias', 'encoder.layer.1.attention.self.query.weight', 'encoder.layer.1.attention.self.query.bias', 'encoder.layer.1.attention.self.key.weight', 'encoder.layer.1.attention.self.key.bias', 'encoder.layer.1.attention.self.value.weight', 'encoder.layer.1.attention.self.value.bias', 'encoder.layer.1.attention.self.query_global.weight', 'encoder.layer.1.attention.self.query_global.bias', 'encoder.layer.1.attention.self.key_global.weight', 'encoder.layer.1.attention.self.key_global.bias', 'encoder.layer.1.attention.self.value_global.weight', 'encoder.layer.1.attention.self.value_global.bias', 'encoder.layer.1.attention.output.dense.weight', 'encoder.layer.1.attention.output.dense.bias', 'encoder.layer.1.attention.output.LayerNorm.weight', 'encoder.layer.1.attention.output.LayerNorm.bias', 'encoder.layer.1.intermediate.dense.weight', 'encoder.layer.1.intermediate.dense.bias', 'encoder.layer.1.output.dense.weight', 'encoder.layer.1.output.dense.bias', 'encoder.layer.1.output.LayerNorm.weight', 'encoder.layer.1.output.LayerNorm.bias', 'encoder.layer.2.attention.self.query.weight', 'encoder.layer.2.attention.self.query.bias', 'encoder.layer.2.attention.self.key.weight', 'encoder.layer.2.attention.self.key.bias', 'encoder.layer.2.attention.self.value.weight', 'encoder.layer.2.attention.self.value.bias', 'encoder.layer.2.attention.self.query_global.weight', 'encoder.layer.2.attention.self.query_global.bias', 'encoder.layer.2.attention.self.key_global.weight', 'encoder.layer.2.attention.self.key_global.bias', 'encoder.layer.2.attention.self.value_global.weight', 'encoder.layer.2.attention.self.value_global.bias', 'encoder.layer.2.attention.output.dense.weight', 'encoder.layer.2.attention.output.dense.bias', 'encoder.layer.2.attention.output.LayerNorm.weight', 'encoder.layer.2.attention.output.LayerNorm.bias', 'encoder.layer.2.intermediate.dense.weight', 'encoder.layer.2.intermediate.dense.bias', 'encoder.layer.2.output.dense.weight', 'encoder.layer.2.output.dense.bias', 'encoder.layer.2.output.LayerNorm.weight', 'encoder.layer.2.output.LayerNorm.bias', 'encoder.layer.3.attention.self.query.weight', 'encoder.layer.3.attention.self.query.bias', 'encoder.layer.3.attention.self.key.weight', 'encoder.layer.3.attention.self.key.bias', 'encoder.layer.3.attention.self.value.weight', 'encoder.layer.3.attention.self.value.bias', 'encoder.layer.3.attention.self.query_global.weight', 'encoder.layer.3.attention.self.query_global.bias', 'encoder.layer.3.attention.self.key_global.weight', 'encoder.layer.3.attention.self.key_global.bias', 'encoder.layer.3.attention.self.value_global.weight', 'encoder.layer.3.attention.self.value_global.bias', 'encoder.layer.3.attention.output.dense.weight', 'encoder.layer.3.attention.output.dense.bias', 'encoder.layer.3.attention.output.LayerNorm.weight', 'encoder.layer.3.attention.output.LayerNorm.bias', 'encoder.layer.3.intermediate.dense.weight', 'encoder.layer.3.intermediate.dense.bias', 'encoder.layer.3.output.dense.weight', 'encoder.layer.3.output.dense.bias', 'encoder.layer.3.output.LayerNorm.weight', 'encoder.layer.3.output.LayerNorm.bias', 'encoder.layer.4.attention.self.query.weight', 'encoder.layer.4.attention.self.query.bias', 'encoder.layer.4.attention.self.key.weight', 'encoder.layer.4.attention.self.key.bias', 'encoder.layer.4.attention.self.value.weight', 'encoder.layer.4.attention.self.value.bias', 'encoder.layer.4.attention.self.query_global.weight', 'encoder.layer.4.attention.self.query_global.bias', 'encoder.layer.4.attention.self.key_global.weight', 'encoder.layer.4.attention.self.key_global.bias', 'encoder.layer.4.attention.self.value_global.weight', 'encoder.layer.4.attention.self.value_global.bias', 'encoder.layer.4.attention.output.dense.weight', 'encoder.layer.4.attention.output.dense.bias', 'encoder.layer.4.attention.output.LayerNorm.weight', 'encoder.layer.4.attention.output.LayerNorm.bias', 'encoder.layer.4.intermediate.dense.weight', 'encoder.layer.4.intermediate.dense.bias', 'encoder.layer.4.output.dense.weight', 'encoder.layer.4.output.dense.bias', 'encoder.layer.4.output.LayerNorm.weight', 'encoder.layer.4.output.LayerNorm.bias', 'encoder.layer.5.attention.self.query.weight', 'encoder.layer.5.attention.self.query.bias', 'encoder.layer.5.attention.self.key.weight', 'encoder.layer.5.attention.self.key.bias', 'encoder.layer.5.attention.self.value.weight', 'encoder.layer.5.attention.self.value.bias', 'encoder.layer.5.attention.self.query_global.weight', 'encoder.layer.5.attention.self.query_global.bias', 'encoder.layer.5.attention.self.key_global.weight', 'encoder.layer.5.attention.self.key_global.bias', 'encoder.layer.5.attention.self.value_global.weight', 'encoder.layer.5.attention.self.value_global.bias', 'encoder.layer.5.attention.output.dense.weight', 'encoder.layer.5.attention.output.dense.bias', 'encoder.layer.5.attention.output.LayerNorm.weight', 'encoder.layer.5.attention.output.LayerNorm.bias', 'encoder.layer.5.intermediate.dense.weight', 'encoder.layer.5.intermediate.dense.bias', 'encoder.layer.5.output.dense.weight', 'encoder.layer.5.output.dense.bias', 'encoder.layer.5.output.LayerNorm.weight', 'encoder.layer.5.output.LayerNorm.bias', 'encoder.layer.6.attention.self.query.weight', 'encoder.layer.6.attention.self.query.bias', 'encoder.layer.6.attention.self.key.weight', 'encoder.layer.6.attention.self.key.bias', 'encoder.layer.6.attention.self.value.weight', 'encoder.layer.6.attention.self.value.bias', 'encoder.layer.6.attention.self.query_global.weight', 'encoder.layer.6.attention.self.query_global.bias', 'encoder.layer.6.attention.self.key_global.weight', 'encoder.layer.6.attention.self.key_global.bias', 'encoder.layer.6.attention.self.value_global.weight', 'encoder.layer.6.attention.self.value_global.bias', 'encoder.layer.6.attention.output.dense.weight', 'encoder.layer.6.attention.output.dense.bias', 'encoder.layer.6.attention.output.LayerNorm.weight', 'encoder.layer.6.attention.output.LayerNorm.bias', 'encoder.layer.6.intermediate.dense.weight', 'encoder.layer.6.intermediate.dense.bias', 'encoder.layer.6.output.dense.weight', 'encoder.layer.6.output.dense.bias', 'encoder.layer.6.output.LayerNorm.weight', 'encoder.layer.6.output.LayerNorm.bias', 'encoder.layer.7.attention.self.query.weight', 'encoder.layer.7.attention.self.query.bias', 'encoder.layer.7.attention.self.key.weight', 'encoder.layer.7.attention.self.key.bias', 'encoder.layer.7.attention.self.value.weight', 'encoder.layer.7.attention.self.value.bias', 'encoder.layer.7.attention.self.query_global.weight', 'encoder.layer.7.attention.self.query_global.bias', 'encoder.layer.7.attention.self.key_global.weight', 'encoder.layer.7.attention.self.key_global.bias', 'encoder.layer.7.attention.self.value_global.weight', 'encoder.layer.7.attention.self.value_global.bias', 'encoder.layer.7.attention.output.dense.weight', 'encoder.layer.7.attention.output.dense.bias', 'encoder.layer.7.attention.output.LayerNorm.weight', 'encoder.layer.7.attention.output.LayerNorm.bias', 'encoder.layer.7.intermediate.dense.weight', 'encoder.layer.7.intermediate.dense.bias', 'encoder.layer.7.output.dense.weight', 'encoder.layer.7.output.dense.bias', 'encoder.layer.7.output.LayerNorm.weight', 'encoder.layer.7.output.LayerNorm.bias', 'encoder.layer.8.attention.self.query.weight', 'encoder.layer.8.attention.self.query.bias', 'encoder.layer.8.attention.self.key.weight', 'encoder.layer.8.attention.self.key.bias', 'encoder.layer.8.attention.self.value.weight', 'encoder.layer.8.attention.self.value.bias', 'encoder.layer.8.attention.self.query_global.weight', 'encoder.layer.8.attention.self.query_global.bias', 'encoder.layer.8.attention.self.key_global.weight', 'encoder.layer.8.attention.self.key_global.bias', 'encoder.layer.8.attention.self.value_global.weight', 'encoder.layer.8.attention.self.value_global.bias', 'encoder.layer.8.attention.output.dense.weight', 'encoder.layer.8.attention.output.dense.bias', 'encoder.layer.8.attention.output.LayerNorm.weight', 'encoder.layer.8.attention.output.LayerNorm.bias', 'encoder.layer.8.intermediate.dense.weight', 'encoder.layer.8.intermediate.dense.bias', 'encoder.layer.8.output.dense.weight', 'encoder.layer.8.output.dense.bias', 'encoder.layer.8.output.LayerNorm.weight', 'encoder.layer.8.output.LayerNorm.bias', 'encoder.layer.9.attention.self.query.weight', 'encoder.layer.9.attention.self.query.bias', 'encoder.layer.9.attention.self.key.weight', 'encoder.layer.9.attention.self.key.bias', 'encoder.layer.9.attention.self.value.weight', 'encoder.layer.9.attention.self.value.bias', 'encoder.layer.9.attention.self.query_global.weight', 'encoder.layer.9.attention.self.query_global.bias', 'encoder.layer.9.attention.self.key_global.weight', 'encoder.layer.9.attention.self.key_global.bias', 'encoder.layer.9.attention.self.value_global.weight', 'encoder.layer.9.attention.self.value_global.bias', 'encoder.layer.9.attention.output.dense.weight', 'encoder.layer.9.attention.output.dense.bias', 'encoder.layer.9.attention.output.LayerNorm.weight', 'encoder.layer.9.attention.output.LayerNorm.bias', 'encoder.layer.9.intermediate.dense.weight', 'encoder.layer.9.intermediate.dense.bias', 'encoder.layer.9.output.dense.weight', 'encoder.layer.9.output.dense.bias', 'encoder.layer.9.output.LayerNorm.weight', 'encoder.layer.9.output.LayerNorm.bias', 'encoder.layer.10.attention.self.query.weight', 'encoder.layer.10.attention.self.query.bias', 'encoder.layer.10.attention.self.key.weight', 'encoder.layer.10.attention.self.key.bias', 'encoder.layer.10.attention.self.value.weight', 'encoder.layer.10.attention.self.value.bias', 'encoder.layer.10.attention.self.query_global.weight', 'encoder.layer.10.attention.self.query_global.bias', 'encoder.layer.10.attention.self.key_global.weight', 'encoder.layer.10.attention.self.key_global.bias', 'encoder.layer.10.attention.self.value_global.weight', 'encoder.layer.10.attention.self.value_global.bias', 'encoder.layer.10.attention.output.dense.weight', 'encoder.layer.10.attention.output.dense.bias', 'encoder.layer.10.attention.output.LayerNorm.weight', 'encoder.layer.10.attention.output.LayerNorm.bias', 'encoder.layer.10.intermediate.dense.weight', 'encoder.layer.10.intermediate.dense.bias', 'encoder.layer.10.output.dense.weight', 'encoder.layer.10.output.dense.bias', 'encoder.layer.10.output.LayerNorm.weight', 'encoder.layer.10.output.LayerNorm.bias', 'encoder.layer.11.attention.self.query.weight', 'encoder.layer.11.attention.self.query.bias', 'encoder.layer.11.attention.self.key.weight', 'encoder.layer.11.attention.self.key.bias', 'encoder.layer.11.attention.self.value.weight', 'encoder.layer.11.attention.self.value.bias', 'encoder.layer.11.attention.self.query_global.weight', 'encoder.layer.11.attention.self.query_global.bias', 'encoder.layer.11.attention.self.key_global.weight', 'encoder.layer.11.attention.self.key_global.bias', 'encoder.layer.11.attention.self.value_global.weight', 'encoder.layer.11.attention.self.value_global.bias', 'encoder.layer.11.attention.output.dense.weight', 'encoder.layer.11.attention.output.dense.bias', 'encoder.layer.11.attention.output.LayerNorm.weight', 'encoder.layer.11.attention.output.LayerNorm.bias', 'encoder.layer.11.intermediate.dense.weight', 'encoder.layer.11.intermediate.dense.bias', 'encoder.layer.11.output.dense.weight', 'encoder.layer.11.output.dense.bias', 'encoder.layer.11.output.LayerNorm.weight', 'encoder.layer.11.output.LayerNorm.bias', 'pooler.dense.weight', 'pooler.dense.bias'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

`

I would like to ask whether the presence of this warning will affect the results?How can  remove this warning?thinks


[trainer] round numbers in trainer state

This PR rounds very long fractions in trainer state e.g.,

```
{'loss': 14.846837043762207, 'learning_rate': 6e-06, 'epoch': 0.3333333333333333}
```
to:
* epoch 2 decimals
* loss 4 decimals

resulting in:

```
{'loss': 14.8468, 'learning_rate': 6e-06, 'epoch': 0.33}
````

If you want any other small tweaks for me to add please let me know.

Fixes: #9475

@sgugger


Using Huggingface library with DeepSpeed

I'm not completely sure if it's this library problem, but maybe you could help.
Trying to run T5-large from huggingface's library with DeepSpeed library I got a strange result. When I change mode to fp16 training loss is going to be NaN value, as well as some of tensors in model's features output. I'm not sure, can it be a Transformers library fault? Original example that I used utilizes pytorch_pretrained_bert, and it works well.
Training with FP32 does not result any NaN troubles.

I have some code, made out of DeepSpeedExamples code:
https://github.com/exelents/try_t5
If somebody would like to help and try to run it, here is compiled binary dataset:
https://drive.google.com/file/d/1oxCxYCuCWebmaUQ_s9il7EDBkisL7x-_/view?usp=sharing
https://drive.google.com/file/d/1WCzxAnp2bEllbQ0_2d_6hoq5tQjxBFXh/view?usp=sharing


fix(wandb): fix config

# What does this PR do?

Fix an issue introduced with PR #9441.
There was just 2 lines to switch related to wandb config detection.

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [X] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [X] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

@sgugger

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Make doc styler detect lists on rst and better support for Windows

# What does this PR do?

The new lines for the lists in rst files were not actually added because I made a mistake, this PR fixes that. @patrickvonplaten it adds some new lines in the benchmarking files which I think are okay, but let me know if I should write some special code to get the scripts to ignore them.

Also, changed the line that added the new lines before doc special words since it seems to be not working properly on Windows. Let's see if this version is better!

(Failures are because master is red at the time of this PR.)

Fixes #9438


[T5] enable T5 fp16

# What does this PR do?

This PR enables fp16 for T5 models, by clamping hidden states to the max value of the current data type.

As detailed in #9295, T5 produces large (`inf`) activations at 3 places
1. Output of `T5LayerFF`
2. Output of `T5LayerSelfAttention`
3. Output of `T5LayerCrossAttention`

To avoid these `inf` activations this PR clamps the `hidden_states` after above 3 outputs


Update run_glue for do_predict with local test data (#9442)

# What does this PR do?

Currently, run_glue.py cannot use the test set (`do_predict`) unless we give it a GLUE task name.
This PR will allow us to use the local test dataset.

As commented in #9442, I tried to achieve the functionality with only simple changes.

- It still works with only the local train and valid files (in other words, this PR does not break the current operation.).
- If we add `--do_predict` with out adding specific params, we will get an error statement saying that we need either the GLUE task name or the path of the local test file.

Fixes #9442

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@sgugger

Thank you for your kind comments on the issue.
I have tried to keep it simple and hope there is no problem as an example script.


ProphetNetNgramAttention: Number of attention heads

## Information

Model I am using (Bert, XLNet ...): ProphetNet

The ProphetNet Ngram attention layer seem to refer to a wrong number of heads. The `ProphetNetNgramProphetNetSelfAttention` (seems to be a typo by the way, maybe `ProphetNetNgramSelfAttention` would be more appropriate?) is part of the decoder, and therefore I would expect it should contain a number of attention head equal to the configuration parameter `num_decoder_attention_heads `.

However, when instantiated at https://github.com/huggingface/transformers/blob/02e05fb0a532e572b56ba75dad6ba3db625bbdeb/src/transformers/models/prophetnet/modeling_prophetnet.py#L759, it uses the property `num_attention_heads` that equals to the number of **encoder** attention heads.

I assume that the correct value should be `config.num_decoder_attention_heads`. (Luckily?) no issue showed up in most models because pretrained models have the same number of encoder and decoder heads. Looking at the reference implementation, it does seem that the **decoder** number of attention heads is used for the Ngram attention (see https://github.com/microsoft/ProphetNet/blob/1d36bc5c4f334b0ed9b90fdf3a64785c174f5c45/GLGE_baselines/script/prophetnet/ngram_s2s_model.py#L586)

### Who can help

 @patrickvonplaten ? Again not sure since I do not see an owner for ProphetNet

I would be happy to submit a small PR referencing to `config.num_decoder_attention_heads` rather than this property if you agree with this change

Thanks!


[Flax] Adapt Flax models to new structure

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

As discussed in https://github.com/huggingface/transformers/pull/9172, Flax model should get a design that is most similar to PyTorch and thus should use `def setup(...)` instead of `nn.compact(...)`. This PR refactors the model architecture of Bert & Roberta accordingly.

The next step is now to add a general conversion method flax<>pytorch which might require some more follow-up changes to the naming of the weights.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Fixing tests. It seems master changed something in the warnings.

# What does this PR do?

Trying to keep warning tests for now. Should be discarded if it becomes
too hard to maintain.

60 started to trigger a new warning, saying that the input_ids length was longer than model max_length.
I'm not really sure which commit triggered this, but it did not occur in the original PR

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

@patrickvonplaten
@LysandreJik

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


Reformat the TF serving outputs

# What does this PR do?

This PR properly reformat the `serving_output` methods.


dataset not being sent to device when using Trainer (distributed)

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.1.1
- Platform: Linux-4.15.0-99-generic-x86_64-with-glibc2.27
- Python version: 3.8.0
- PyTorch version (GPU?): 1.7.1 (True)
- Tensorflow version (GPU?): 2.4.0 (False)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: distrubuted

### Who can help
 Trainer: @sgugger
 Text Generation /t5: @patrickvonplaten
 examples/seq2seq: @patil-suraj

## Information
The intention is to train a t5 model (preferably as last as possible) in a distributed setting using the HF Trainer. However, when setting the model_parallel to True the training breaks.

related issues might be:
https://github.com/huggingface/transformers/issues/9229
https://github.com/huggingface/transformers/issues/6821

However, do note that the script works perfectly fine training on multiple GPU in a non distributed fashion (setting model_parallel to False).

## To reproduce

I have created a minimal script for reproducing the behavior:

```
import transformers
from transformers import (
    MT5ForConditionalGeneration,
    MT5Model,
    Trainer,
    TrainingArguments,
    T5Tokenizer
)
```

<details>
  <summary> Click to see (minimal) dataset creation </summary>

```
import datasets

# making minimal test for example sake
def make_test_dataset(tokenizer="google/mt5-small"):
    if isinstance(tokenizer, str):
        tokenizer = T5Tokenizer.from_pretrained(tokenizer)

    ds = datasets.load_dataset("dane")

    def __tokenizer_input(batch):
        return tokenizer(batch['text'],
                         padding="max_length",
                         max_length=256,  # actual max is 235
                         truncation=True)

    def __tokenizer_output(batch):
        tok = tokenizer(batch['text'],
                        padding="max_length",
                        max_length=256,
                        truncation=True)
        tok["labels"] = tok.pop("input_ids")
        return tok

    # filter out empty strings (bug reported and fixed)
    ds = ds.filter(lambda batch: bool(batch["text"]))

    # tokenize both datasets (eos:   </s> is added by tokenizer)
    ds = ds.map(__tokenizer_input, batched=True, batch_size=len(ds))
    ds = ds.map(__tokenizer_output, batched=True, batch_size=len(ds))

    return ds

dataset = make_test_dataset()
dataset.set_format(type='torch',
                   columns=['input_ids',
                            'attention_mask',
                            'labels'])
```

</details>

```
model = MT5ForConditionalGeneration.from_pretrained("google/mt5-small") #using a small model for example
training_args = TrainingArguments(
    output_dir='./results',
    num_train_epochs=1,
    logging_dir='./logs',
    evaluation_strategy="epoch",
    model_parallel=True # work fine when set to False
)
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=dataset["train"],
    eval_dataset=dataset["test"]
)
trainer.remove_callback(transformers.integrations.WandbCallback) # removing wandb for conv.
trainer.train()
```

Results:

```
RuntimeError: Input, output and indices must be on the current device
```

<details>
  <summary> Click to see full traceback </summary>

```
  RuntimeError                              Traceback (most recent call last)
  ~/github/EDP-Efficient-Danish-Preprocessing/tmp.py in
       64
       65 trainer.remove_callback(transformers.integrations.WandbCallback)
  ---> 66 trainer.train()

  ~/.Envs/EDP/lib/python3.8/site-packages/transformers/trainer.py in train(self, model_path, trial)
      797                         tr_loss += self.training_step(model, inputs)
      798                 else:
  --> 799                     tr_loss += self.training_step(model, inputs)
      800                 self._total_flos += self.floating_point_ops(inputs)
      801

  ~/.Envs/EDP/lib/python3.8/site-packages/transformers/trainer.py in training_step(self, model, inputs)
     1137                 loss = self.compute_loss(model, inputs)
     1138         else:
  -> 1139             loss = self.compute_loss(model, inputs)
     1140
     1141         if self.args.n_gpu > 1:

  ~/.Envs/EDP/lib/python3.8/site-packages/transformers/trainer.py in compute_loss(self, model, inputs)
     1161         Subclass and override for custom behavior.
     1162         """
  -> 1163         outputs = model(**inputs)
     1164         # Save past state if it exists
     1165         # TODO: this needs to be fixed and made cleaner later.

  ~/.Envs/EDP/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
      725             result = self._slow_forward(*input, **kwargs)
      726         else:
  --> 727             result = self.forward(*input, **kwargs)
      728         for hook in itertools.chain(
      729                 _global_forward_hooks.values(),

  ~/.Envs/EDP/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py in forward(self, input_ids, attention_mask, decoder_input_ids, decoder_attention_mask, encoder_outputs, past_key_values, head_mask, inputs_embeds, decoder_inputs_embeds, labels, use_cache, output_attentions, output_hidden_states, return_dict)
     1422         if encoder_outputs is None:
     1423             # Convert encoder inputs in embeddings if needed
  -> 1424             encoder_outputs = self.encoder(
     1425                 input_ids=input_ids,
     1426                 attention_mask=attention_mask,

  ~/.Envs/EDP/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
      725             result = self._slow_forward(*input, **kwargs)
      726         else:
  --> 727             result = self.forward(*input, **kwargs)
      728         for hook in itertools.chain(
      729                 _global_forward_hooks.values(),

  ~/.Envs/EDP/lib/python3.8/site-packages/transformers/models/t5/modeling_t5.py in forward(self, input_ids, attention_mask, encoder_hidden_states, encoder_attention_mask, inputs_embeds, head_mask, past_key_values, use_cache, output_attentions, output_hidden_states, return_dict)
      858         if inputs_embeds is None:
      859             assert self.embed_tokens is not None, "You have to initialize the model with valid token embeddings"
  --> 860             inputs_embeds = self.embed_tokens(input_ids)
      861
      862         batch_size, seq_length = input_shape

  ~/.Envs/EDP/lib/python3.8/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
      725             result = self._slow_forward(*input, **kwargs)
      726         else:
  --> 727             result = self.forward(*input, **kwargs)
      728         for hook in itertools.chain(
      729                 _global_forward_hooks.values(),

  ~/.Envs/EDP/lib/python3.8/site-packages/torch/nn/modules/sparse.py in forward(self, input)
      122
      123     def forward(self, input: Tensor) -> Tensor:
  --> 124         return F.embedding(
      125             input, self.weight, self.padding_idx, self.max_norm,
      126             self.norm_type, self.scale_grad_by_freq, self.sparse)

  ~/.Envs/EDP/lib/python3.8/site-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
     1850         # remove once script supports set_grad_enabled
     1851         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
  -> 1852     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
     1853
     1854

  RuntimeError: Input, output and indices must be on the current device
```
</details>

## Expected behavior
A training model

PS: I personally found that the model_parallel name was slightly confusing. I assume a more fitting name would be model_distributed (but this is a minor thing)

Thanks for great work


request for run_text_classification.py

# 🚀 Feature request

There is a run_tf_text_classification.py file under text_classification examples, but no run_text_classification.py.


Makes HfArgumentParser compatible with Python 3.9

Python 3.9 changed the format of the string serialization of `typing.Optional`.

For example, `str(typing.Optional[str])` is `typing.Union[str, NoneType]` in python 3.8 and `typing.Optional[str]` in Python 3.9.


Fix TF s2s models

# What does this PR do?

This PR aims to fix the Seq2Seq models in order to make them able to be served through TF Serving.  The problem is stated by @patrickvonplaten in #9313. The reason why it failed was because we use a model as a layer in the `TFXXXForConditionalGeneration` models. The tracing mechanism of TensorFlow when building a graph calls one by one all the layers for building the graph. In order to know what are the inputs needed by each layer, the tracing mechanism check if a layer has a custom input signature, if not, it takes as default a signature where only the first argument is mandatory. Here stand the problem, the Seq2Seq models needs have two mandatory arguments (`input_ids` and `decoder_input_ids` or `inputs_embeds` and `decoder_inputs_embeds`) and then the tracing fails.

The fix to this problem is to manually set the expected input signature of the base model when instantiating it in `__init__`. To be harmonized with the required serving, the same signature is used.


rename "gpu" --> "device"

# What does this PR do?

Rename the arg names from "per_gpu" to "per_device" such that it aligns with the instruction in the readme  https://github.com/huggingface/transformers/tree/master/examples/text-classification#xnli


Improve LayoutLM

# What does this PR do?

- [x] Improve documentation of `LayoutLM`, explaining how people can normalize bounding boxes before passing them to the model, add links to the various datasets on which the model achieves state-of-the-art results, add code examples in the documentation for the various models
- [x] Add notebook to the list of community notebooks showcasing how to fine-tune `LayoutLMForTokenClassification` on the [FUNSD](https://guillaumejaume.github.io/FUNSD/) dataset (on which the model achieves SOTA results)
- [x] Add integration tests, which confirm that the model outputs the same tensors as the original implementation on the same input data
- [x] Add `LayoutLMForSequenceClassification`, which makes it possible to fine-tune LayoutLM for document image classification tasks (such as the [RVL-CLIP dataset](https://www.cs.cmu.edu/~aharley/rvl-cdip/)), extra tests included.

Fixes the following issues:
- #9228
- #9097
- #8866
- #8524

## Who can review?

@LysandreJik, @patrickvonplaten, @sgugger


[trainer] fractional epoch

Running
```
export BS=20; rm -r output_dir; CUDA_VISIBLE_DEVICES=0,1 PYTHONPATH=../../src USE_TF=0   ./finetune_trainer.py --model_name_or_path t5-small --output_dir output_dir --adam_eps 1e-06 --data_dir wmt_en_ro --do_eval --do_predict --do_train --evaluation_strategy=steps --freeze_embeds --label_smoothing 0.1 --learning_rate 3e-5 --logging_first_step --logging_steps 1000 --max_source_length 128 --max_target_length 128 --num_train_epochs 1 --overwrite_output_dir --per_device_eval_batch_size $BS --per_device_train_batch_size $BS --predict_with_generate --eval_steps 25000  --sortish_sampler --task translation_en_to_ro --test_max_target_length 128 --val_max_target_length 128 --warmup_steps 5 --n_train 100 --n_val 100 --n_test 100  --fp16 --save_steps 1
```
on master, gives:
```
{'loss': 14.846837043762207, 'learning_rate': 6e-06, 'epoch': 0.3333333333333333}
```
epoch can't be fractional.


Fast imports part 3

# What does this PR do?

This is the last PR to make the import of transformers and defer the imports of torch/tensorflow to when is necessary. It does the same work as #9446 but in each itnermediate init, so that
```
from transformers import BertModel
```
only imports torch and not TensorFlow (ans is thus very fast).

The templates are adapted to the new init format, so users adding models don't have to worry about this.
In passing, I noticed that the `tokenization_utils_base` was importing everything at init, so I deferred imports there to only do them when necessary. There might be a few places like this left, but we can address those later on.

Fixes #8733


[Generation Tests] Small speed-up by just generating two tokens

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

@LysandreJik @sgugger I originally thought that the PR: https://github.com/huggingface/transformers/commit/c89f1bc92e340600bde526b7ff54ad692b4e48c9 made the PyTorch tests much slower, but after checking the time of `run_tests_torch` in 10+ merges to master after and before this commit, I noticed that the PR didn't really affect the testing time of PyTorch.

The testing time varies quite a bit, but it seemed on average to be a bit higher after the merged PR, so in this PR I want to reduce  the testing time for generation a bit. The generation length is reduced by one which halves the testing time of all generation tests by 30% without any loss in testing coverage / cases. Generating two tokens is enough => the first token can be generated without `past_key_values`, but the second token has to be generated with `past_key_values` if `use_cache` is enabled and all generation steps following this one can only be the same. So we should always test at least two tokens, but don't really need to test more in general generation tests that apply to all models.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


[Generation] Fix bug for manual decoder_input_ids + warning message

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Some improvements on the design of how `decoder_input_ids` are extracted that solve: https://github.com/huggingface/transformers/issues/9400
Also adds a nicer warning to prevent non-understandable errors as shown in: https://github.com/huggingface/transformers/issues/9464

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


model.generate() has the same speed on CPU and GPU

Hi, I find that model.generate() of BART and T5 has roughly the same running speed when running on CPU and GPU. Why doesn't GPU give faster speed? Thanks!

## Environment info
- `transformers` version: 4.1.1
- Python version: 3.6
- PyTorch version (GPU?): 1.3.1
- Using GPU in script?: yes

### Who can help
 TextGeneration: @TevenLeScao
 Bart: @patrickvonplaten
 Pegasus: @patrickvonplaten
 mBART: @patrickvonplaten
 T5: @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): BART and T5
## To reproduce
```python
import time
from transformers import BartTokenizer, BartForConditionalGeneration

device = 'cpu'

# change to GPU
# device = 'cuda:0'

text_to_summarize = "My friends are cool but they eat too many carbs."

tokenizer = BartTokenizer.from_pretrained('facebook/bart-base')
model = BartForConditionalGeneration.from_pretrained('facebook/bart-base')

inputs = tokenizer(text_to_summarize, return_tensors='pt')

inputs = inputs['input_ids'].to(device)
model = model.to(device)

start = time.time()
summary_ids = model.generate(inputs)
print("Time spent (s): ", time.time() - start)
```
## Expected behavior
I expected running on GPU should give me much faster speed. But running on GPU gave me roughly the same speed as CPU, both around 0.3s in this case.


max_target length for question answering system

Please could you tell the max target length for question answering systems?

I was trying and it does not work if the target length is more than 47

this is the   [notebook](https://colab.research.google.com/drive/1JzsuPb68L-G4nsMXu57XkwbmzAVrizSS?usp=sharing)


Cannot Evaluate While Training Using the Trainer

@sgugger

## Environment info

- `transformers` version: 4.0.0
- Platform: AWS Amazon Linux
- Python version: 3.6.10
- PyTorch version (GPU?): 1.5.1
- Tensorflow version (GPU?): N/A
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

## Information

Model I am using (Bert, XLNet ...): Third Party Model ( Routing Transformer)

The problem arises when using:
* [ ] my own modified scripts:
I am using a custom Trainer to train the third party model. The training loops runs smoothly without evaluation. But as soon as I try to do evaluation while training, it stops training after a couple of evaluations even though it has not reached the max_steps. I copy my args and trainer below.  I understand in previous versions there was a evaluate_during_training flag that some have suggested as a fix until Sep 20. But that flag doesn't seem to exist anymore. Any help or pointers would be appreciated.

The tasks I am working on is:
* [ ] my own task or dataset: Masked Language Modeling

## To reproduce

Steps to reproduce the behavior:

```
custom_args = MLMArguments(
        output_dir='../models/',
        mask_ratio=0.2,
        do_train=True,
        do_eval=True,
        max_steps=2000000,
        save_steps=50,
        logging_steps=5,
        per_device_train_batch_size=1,
        per_device_eval_batch_size=1,
        logging_dir = '../logs/',
        evaluation_strategy="steps",
        prediction_loss_only=True

)

checkpoint_callback=TrainerCallback()
tb_callback = TensorBoardCallback()

custom_trainer = MLMTrainer(
        rt_model,
        args=custom_args,
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        callbacks=[checkpoint_callback, tb_callback]

)

custom_trainer.train()
```

Step | Training Loss | Validation Loss
-- | -- | --
5 | 10.033914 | 10.017973
10 | 10.028201 | 9.935969

and at this point it prints the output and stops training.


# 🚀 Have RAG return generator cross-attentions when output_attentions=True

This feature request is for the RAG code to be modified so that if `output_attentions=True`, it returns the generator's cross-attentions in addition to the attentions it already returns.

## Motivation

I'm interested in extracting the generator's attentions from a RAG generator model. Currently, `transformers` allows you to extract the generator's encoder attentions and decoder attentions, but not it's cross attentions. For example, inside `modeling_rag.py`, the return objects such as [RetrievAugLMMarginOutput](https://github.com/huggingface/transformers/blob/master/src/transformers/models/rag/modeling_rag.py#L38), have fields for these other attentions, but not the cross-attentions.

Because both T5 and BART can output cross-attentions, I think they could simply propagate up through the RAG code. Is there a reason this isn't already the case? Or could I do a PR to include the cross attentions along with the other attentions in the model output?

## Your contribution

On my own fork of `transformers`, I've already added this feature and would happily submit a PR!


Unable to train sequence classification task using TFTrainer

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.1.1
- Platform: Linux-4.15.0-123-generic-x86_64-with-debian-buster-sid
- Python version: 3.7.9
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.4.0 (False)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 -->
Trainer: @sgugger
Tensorflow:  @jplu

## Information

Model I am using (Bert, XLNet ...): distilbert-base-cased

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The following is how I load the model and the trainer
```{python}
from sklearn.metrics import accuracy_score, precision_recall_fscore_support
from datasets import load_from_disk
import tensorflow as tf
from transformers import TFAutoModelForSequenceClassification, AutoTokenizer
from transformers import TFTrainingArguments, TFTrainer

# Load dataset
def load_dataset(data_dir, split="train", batch_size=32, shuffle=100):
    dataset = load_from_disk(data_dir)[split]

    label_type = tf.int32
    input_names = ["input_ids", "attention_mask", "token_type_ids"]

    def gen():
        for ex in dataset:
            d = {k: v for k, v in ex.items() if v is not None}
            label = d.pop("tag")
            yield (d, label)

    tf_dataset = tf.data.Dataset.from_generator(
        gen,
        ({k: tf.int32 for k in input_names}, label_type),
        ({k: tf.TensorShape([None]) for k in input_names}, tf.TensorShape([])),
    )
    tf_dataset = tf_dataset.apply(tf.data.experimental.assert_cardinality(len(dataset)))
    return tf_dataset

# Load the model
def load_model(name="distilbert-base-cased", num_labels=11, learning_rate=3e-5):
    tokenizer = AutoTokenizer.from_pretrained(name)
    tokenizer.add_special_tokens({"bos_token": "<s>", "eos_token": "</s>"})
    model = TFAutoModelForSequenceClassification.from_pretrained(
        name, num_labels=num_labels
    )
    model.resize_token_embeddings(len(tokenizer))
    optimizer = tf.keras.optimizers.Adam(learning_rate=learning_rate)
    loss = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
    model.compile(optimizer=optimizer, loss=loss)
    model.summary()
    return model

#metrics
def compute_metrics(pred):
    labels = pred.label_ids
    predictions = pred.predictions.argmax(-1)
    precision, recall, f1, _ = precision_recall_fscore_support(
        labels, predictions, average="weighted"
    )
    acc = accuracy_score(labels, predictions)
    return {"accuracy": acc, "f1": f1, "precision": precision, "recall": recall}

# train model
def train_model(
    model_args, data_dir, model_dir, logs_dir, batch_size=32, num_epochs=10
):
    training_args = TFTrainingArguments(
        output_dir=model_dir,
        num_train_epochs=num_epochs,
        do_train=True,
        per_device_train_batch_size=batch_size,
        per_device_eval_batch_size=batch_size * 2,
        evaluation_strategy="steps",
        warmup_steps=500,
        weight_decay=0.01,
        logging_dir=logs_dir,
        dataloader_num_workers=15,
    )
    datasets = {
        "train": load_dataset(data_dir=data_dir, split="train", batch_size=batch_size),
        "val": load_dataset(
            data_dir=data_dir, split="validation", batch_size=batch_size
        ),
    }

    model = load_model(**model_args)

    trainer = TFTrainer(
        model=model,
        args=training_args,
        train_dataset=datasets["train"],
        eval_dataset=datasets["val"],
        compute_metrics=compute_metrics,
    )
    trainer.train()
    return trainer
```
The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)
A multi-class classification task to classify a sentence into one of 11 known categories

## To reproduce

Steps to reproduce the behavior:

1. A classification task with more the 2 categories - i.e. num_labels > 2.
2. Use pretrained distill bert model for sequence classification
3. Load the dataset and finetune the model with TFTrainer.

<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

Here is the stack trace from the error.
```
ValueError: in user code:

    /root/.local/share/virtualenvs/ai-dialogue-acts-classifier-ZQsBEC0q/lib/python3.7/site-packages/transformers/trainer_tf.py:678 distributed_training_steps  *
        self.args.strategy.run(self.apply_gradients, inputs)
    /root/.local/share/virtualenvs/ai-dialogue-acts-classifier-ZQsBEC0q/lib/python3.7/site-packages/transformers/trainer_tf.py:641 apply_gradients  *
        self.optimizer.apply_gradients(list(zip(gradients, self.model.trainable_variables)))
    /root/.local/share/virtualenvs/ai-dialogue-acts-classifier-ZQsBEC0q/lib/python3.7/site-packages/transformers/optimization_tf.py:232 apply_gradients  *
        return super(AdamWeightDecay, self).apply_gradients(zip(grads, tvars), name=name, **kwargs)
    /root/.local/share/virtualenvs/ai-dialogue-acts-classifier-ZQsBEC0q/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:604 apply_gradients  **
        self._create_all_weights(var_list)
    /root/.local/share/virtualenvs/ai-dialogue-acts-classifier-ZQsBEC0q/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:783 _create_all_weights
        self._create_slots(var_list)
    /root/.local/share/virtualenvs/ai-dialogue-acts-classifier-ZQsBEC0q/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/adam.py:127 _create_slots
        self.add_slot(var, 'm')
    /root/.local/share/virtualenvs/ai-dialogue-acts-classifier-ZQsBEC0q/lib/python3.7/site-packages/tensorflow/python/keras/optimizer_v2/optimizer_v2.py:844 add_slot
        .format(strategy, var))

    ValueError: Trying to create optimizer slot variable under the scope for tf.distribute.Strategy (<tensorflow.python.distribute.one_device_strategy.OneDeviceStrategy object at 0x7efd64765bd0>), which is different from the scope used for the original variable (<tf.Variable 'tf_distil_bert_for_sequence_classification/distilbert/embeddings/tf_distil_bert_for_sequence_classification/distilbert/embeddings/word_embeddings/weight:0' shape=(28998, 768) dtype=float32, numpy=
    array([[-0.02513016, -0.03304445, -0.00243959, ..., -0.01084836,
            -0.04682418, -0.00948554],
           [-0.00482445, -0.02148623, -0.00871447, ..., -0.02602929,
            -0.03786189, -0.02410287],
           [-0.01653061, -0.01786226,  0.00105964, ..., -0.01637051,
            -0.03567044, -0.03141942],
           ...,
           [ 0.01190545, -0.02329331, -0.02250608, ..., -0.02713599,
            -0.04355597,  0.00010529],
           [ 0.00688736,  0.02267248,  0.02263871, ..., -0.00735895,
            -0.00814128,  0.00426289],
           [ 0.00320692, -0.0061747 ,  0.01624888, ...,  0.00641411,
             0.00060032,  0.01258053]], dtype=float32)>). Make sure the slot variables are created under the same strategy scope. This may happen if you're restoring from a checkpoint outside the scope
```

## Expected behavior

<!-- A clear and concise description of what you would expect to happen. -->
The same model trains successfully when trained as a tf.keras model with a batched tfdataset.
```
# modified training code to use the keras model instance that trains the model successfully.
def train_model(
    model_args, data_dir, model_dir, logs_dir, batch_size=32, num_epochs=10
):
    training_args = TFTrainingArguments(
        output_dir=model_dir,
        num_train_epochs=num_epochs,
        do_train=True,
        per_device_train_batch_size=batch_size,
        per_device_eval_batch_size=batch_size * 2,
        evaluation_strategy="steps",
        warmup_steps=500,
        weight_decay=0.01,
        logging_dir=logs_dir,
        dataloader_num_workers=15,
    )
    datasets = {
        "train": load_dataset(data_dir=data_dir, split="train", batch_size=batch_size),
        "val": load_dataset(
            data_dir=data_dir, split="validation", batch_size=batch_size
        ),
    }

    model = load_model(**model_args)

    history = model.fit(datasets["train"].batch(32), verbose=1)
    return history
```


RuntimeError when running Reformer model

## Environment info

- `transformers` version: 2.10.0
- Platform: Linux-5.4.0-1034-aws-x86_64-with-debian-buster-sid
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no

### Who can help
 --> @patrickvonplaten

## Information

Model I am using (Bert, XLNet ...): Reformer

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Run the example code from [here](https://huggingface.co/google/reformer-crime-and-punishment?text=My+name+is+Julien+and+I+like+to):
```
model = ReformerModelWithLMHead.from_pretrained("google/reformer-crime-and-punishment")
tok = ReformerTokenizer.from_pretrained("google/reformer-crime-and-punishment")
tok.decode(model.generate(tok.encode("A few months later", return_tensors="pt"), do_sample=True,temperature=0.7, max_length=100)[0])
```
<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

RuntimeError: Overflow when unpacking long (more details below)

```
<ipython-input-33-0a824540b4e0> in <module>
      2 tok = ReformerTokenizer.from_pretrained("google/reformer-crime-and-punishment")
      3 tok.decode(model.generate(tok.encode("A few months later", return_tensors="pt"),
----> 4                           do_sample=True,temperature=0.7)[0])

~/anaconda3/envs/Reformer/lib/python3.6/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     13         def decorate_context(*args, **kwargs):
     14             with self:
---> 15                 return func(*args, **kwargs)
     16         return decorate_context
     17

~/anaconda3/envs/Reformer/lib/python3.6/site-packages/transformers/modeling_utils.py in generate(self, input_ids, max_length, min_length, do_sample, early_stopping, num_beams, temperature, top_k, top_p, repetition_penalty, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, length_penalty, no_repeat_ngram_size, num_return_sequences, attention_mask, decoder_start_token_id, use_cache, **model_specific_kwargs)
   1179                 attention_mask=attention_mask,
   1180                 use_cache=use_cache,
-> 1181                 model_specific_kwargs=model_specific_kwargs,
   1182             )
   1183

~/anaconda3/envs/Reformer/lib/python3.6/site-packages/transformers/modeling_utils.py in _generate_no_beam_search(self, input_ids, cur_len, max_length, min_length, do_sample, temperature, top_k, top_p, repetition_penalty, no_repeat_ngram_size, bad_words_ids, bos_token_id, pad_token_id, eos_token_id, decoder_start_token_id, batch_size, encoder_outputs, attention_mask, use_cache, model_specific_kwargs)
   1221             )
   1222
-> 1223             outputs = self(**model_inputs)
   1224             next_token_logits = outputs[0][:, -1, :]
   1225

~/anaconda3/envs/Reformer/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

~/anaconda3/envs/Reformer/lib/python3.6/site-packages/transformers/modeling_reformer.py in forward(self, input_ids, position_ids, attention_mask, head_mask, inputs_embeds, num_hashes, labels, do_output_hidden_states, do_output_attentions)
   1738             num_hashes=num_hashes,
   1739             do_output_hidden_states=do_output_hidden_states,
-> 1740             do_output_attentions=do_output_attentions,
   1741         )
   1742

~/anaconda3/envs/Reformer/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

~/anaconda3/envs/Reformer/lib/python3.6/site-packages/transformers/modeling_reformer.py in forward(self, input_ids, attention_mask, position_ids, head_mask, inputs_embeds, num_hashes, do_output_hidden_states, do_output_attentions)
   1588             num_hashes=num_hashes,
   1589             do_output_hidden_states=do_output_hidden_states,
-> 1590             do_output_attentions=do_output_attentions,
   1591         )
   1592         sequence_output = encoder_outputs.hidden_states

~/anaconda3/envs/Reformer/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

~/anaconda3/envs/Reformer/lib/python3.6/site-packages/transformers/modeling_reformer.py in forward(self, hidden_states, attention_mask, head_mask, num_hashes, do_output_hidden_states, do_output_attentions)
   1324             all_attentions,
   1325             do_output_hidden_states,
-> 1326             do_output_attentions,
   1327         )
   1328

~/anaconda3/envs/Reformer/lib/python3.6/site-packages/transformers/modeling_reformer.py in forward(ctx, hidden_states, layers, attention_mask, head_mask, num_hashes, all_hidden_states, all_attentions, do_output_hidden_states, do_output_attentions)
   1220                 head_mask=layer_head_mask,
   1221                 num_hashes=num_hashes,
-> 1222                 do_output_attentions=do_output_attentions,
   1223             )
   1224             attn_output = layer_outputs.attn_output

~/anaconda3/envs/Reformer/lib/python3.6/site-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

~/anaconda3/envs/Reformer/lib/python3.6/site-packages/transformers/modeling_reformer.py in forward(***failed resolving arguments***)
   1111             # for dropout and save seed for forward fn in backward
   1112             # to have correct dropout
-> 1113             self._init_feed_forward_seed()
   1114             # Y_2 = X_2 + g(Y_1)
   1115             hidden_states = hidden_states + self.feed_forward(attn_output)

~/anaconda3/envs/Reformer/lib/python3.6/site-packages/transformers/modeling_reformer.py in _init_feed_forward_seed(self)
   1075         else:
   1076             # CPU
-> 1077             self.feed_forward_seed = int(torch.seed() % sys.maxsize)
   1078             torch.manual_seed(self.feed_forward_seed)
   1079

~/anaconda3/envs/Reformer/lib/python3.6/site-packages/torch/random.py in seed()
     43
     44     if not torch.cuda._is_in_bad_fork():
---> 45         torch.cuda.manual_seed_all(seed)
     46
     47     return seed

~/anaconda3/envs/Reformer/lib/python3.6/site-packages/torch/cuda/random.py in manual_seed_all(seed)
    111             default_generator.manual_seed(seed)
    112
--> 113     _lazy_call(cb)
    114
    115

~/anaconda3/envs/Reformer/lib/python3.6/site-packages/torch/cuda/__init__.py in _lazy_call(callable)
    133 def _lazy_call(callable):
    134     if is_initialized():
--> 135         callable()
    136     else:
    137         # Don't store the actual traceback to avoid memory cycle

~/anaconda3/envs/Reformer/lib/python3.6/site-packages/torch/cuda/random.py in cb()
    109         for i in range(device_count()):
    110             default_generator = torch.cuda.default_generators[i]
--> 111             default_generator.manual_seed(seed)
    112
    113     _lazy_call(cb)

RuntimeError: Overflow when unpacking long
```


[README] Add new models

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Adds LED and BlenderbotSmall to the Readme.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/master/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/master/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors which may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

 albert, bert, XLM: @LysandreJik
 GPT2: @LysandreJik, @patrickvonplaten
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @patrickvonplaten, @TevenLeScao
 Blenderbot, Bart, Marian, Pegasus: @patrickvonplaten
 T5: @patrickvonplaten
 Rag: @patrickvonplaten, @lhoestq
 EncoderDecoder: @patrickvonplaten
 Longformer, Reformer: @patrickvonplaten
 TransfoXL, XLNet: @TevenLeScao, @patrickvonplaten
 examples/seq2seq: @patil-suraj
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 FSMT: @stas00
 -->


UnboundLocalError when generating sequences

## Environment info

- `transformers` version: 4.2.0dev0
- Platform: macOS-11.1-x86_64-i386-64bit
- Python version: 3.8.7
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help

@patrickvonplaten

## Information

Model I am using GPT2LMHeadModel:

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

Generate sequences using with the following snippet:
```python
model.generate(
        input_ids,
        do_sample=False,
        num_beams=beam_width,
        num_return_sequences=beam_width,
        early_stopping=False,
        output_scores=True,
        return_dict_in_generate=True,
)
```

Generate

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Try to generate sequences as mentioned above.

### Traceback

```
Traceback (most recent call last):
  File "/Users/miguelvictor/.virtualenvs/transformers/lib/python3.8/site-packages/uvicorn/protocols/http/h11_impl.py", line 394, in run_asgi
    result = await app(self.scope, self.receive, self.send)
  File "/Users/miguelvictor/.virtualenvs/transformers/lib/python3.8/site-packages/uvicorn/middleware/proxy_headers.py", line 45, in __call__
    return await self.app(scope, receive, send)
  File "/Users/miguelvictor/.virtualenvs/transformers/lib/python3.8/site-packages/fastapi/applications.py", line 199, in __call__
    await super().__call__(scope, receive, send)
  File "/Users/miguelvictor/.virtualenvs/transformers/lib/python3.8/site-packages/starlette/applications.py", line 111, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/Users/miguelvictor/.virtualenvs/transformers/lib/python3.8/site-packages/starlette/middleware/errors.py", line 181, in __call__
    raise exc from None
  File "/Users/miguelvictor/.virtualenvs/transformers/lib/python3.8/site-packages/starlette/middleware/errors.py", line 159, in __call__
    await self.app(scope, receive, _send)
  File "/Users/miguelvictor/.virtualenvs/transformers/lib/python3.8/site-packages/starlette/middleware/cors.py", line 86, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/Users/miguelvictor/.virtualenvs/transformers/lib/python3.8/site-packages/starlette/middleware/cors.py", line 142, in simple_response
    await self.app(scope, receive, send)
  File "/Users/miguelvictor/.virtualenvs/transformers/lib/python3.8/site-packages/starlette/exceptions.py", line 82, in __call__
    raise exc from None
  File "/Users/miguelvictor/.virtualenvs/transformers/lib/python3.8/site-packages/starlette/exceptions.py", line 71, in __call__
    await self.app(scope, receive, sender)
  File "/Users/miguelvictor/.virtualenvs/transformers/lib/python3.8/site-packages/starlette/routing.py", line 566, in __call__
    await route.handle(scope, receive, send)
  File "/Users/miguelvictor/.virtualenvs/transformers/lib/python3.8/site-packages/starlette/routing.py", line 227, in handle
    await self.app(scope, receive, send)
  File "/Users/miguelvictor/.virtualenvs/transformers/lib/python3.8/site-packages/starlette/routing.py", line 41, in app
    response = await func(request)
  File "/Users/miguelvictor/.virtualenvs/transformers/lib/python3.8/site-packages/fastapi/routing.py", line 201, in app
    raw_response = await run_endpoint_function(
  File "/Users/miguelvictor/.virtualenvs/transformers/lib/python3.8/site-packages/fastapi/routing.py", line 150, in run_endpoint_function
    return await run_in_threadpool(dependant.call, **values)
  File "/Users/miguelvictor/.virtualenvs/transformers/lib/python3.8/site-packages/starlette/concurrency.py", line 34, in run_in_threadpool
    return await loop.run_in_executor(None, func, *args)
  File "/usr/local/opt/python@3.8/Frameworks/Python.framework/Versions/3.8/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "./server.py", line 47, in doSampleGPT2
    results = sampleGPT2v2(model=model, tokenizer=tokenizer, sequence=src)
  File "./ccompletion/samplers/gpt2sampler.py", line 151, in sampleGPT2v2
    outputs = model.generate(
  File "/Users/miguelvictor/.virtualenvs/transformers/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "/Users/miguelvictor/Projects/transformers/src/transformers/generation_utils.py", line 943, in generate
    return self.beam_search(
  File "/Users/miguelvictor/Projects/transformers/src/transformers/generation_utils.py", line 1655, in beam_search
    input_ids, beam_scores, next_tokens, next_indices, pad_token_id=pad_token_id, eos_token_id=eos_token_id
UnboundLocalError: local variable 'next_tokens' referenced before assignment
```

## Expected behavior

No errors raised.


FileNotFoundError when instantiating RagRetriever

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.1.1
- Platform: Linux-5.4.0-58-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.9
- PyTorch version (GPU?): 1.7.1 (True)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten, @lhoestq

## Information

I am trying to use RAG but I am having issues downloading the compressed index.

## To reproduce

```python
from transformers import RagRetriever
retriever = RagRetriever.from_pretrained('facebook/rag-sequence-nq', dataset='wiki_dpr', index_name='compressed')
```
Which results in:
```python
FileNotFoundError: Couldn't find file at https://storage.googleapis.com/huggingface-nlp/cache/datasets/wiki_dpr/psgs_w100.nq.compressed/0.0.0/psgs_w100.nq.IVF4096_HNSW128_PQ128-IP-train.faiss
```


Fix scatter import

Scatter is wrongly spelled