Long BERT TypeError: forward() takes from 2 to 4 positional arguments but 7 were given
Author: paulthemagno
Pull request: no
Labels: name: wontfix

I'm having an issue on the pretraining of a BERT-like model. I used the following function twice: the first time with [bert-base-multilingual-cased](https://storage.googleapis.com/bert_models/2018_11_23/multi_cased_L-12_H-768_A-12.zip) and the second time with a simil version, but more efficient for **long documents**, exploiting the class [LongformerSelfAttention](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_longformer.py#L100) to make the normal BERT into a **LongBERT**.

```python
def pretrain_and_evaluate(args, model, tokenizer, eval_only, model_path):
    val_dataset = TextDataset(tokenizer=tokenizer,
                              file_path=args.val_datapath,
                              block_size=tokenizer.max_len)
    if eval_only:
        train_dataset = val_dataset
    else:
        logger.info(f'Loading and tokenizing training data is usually slow: {args.train_datapath}')
        train_dataset = TextDataset(tokenizer=tokenizer,
                                    file_path=args.train_datapath,
                                    block_size=tokenizer.max_len)

    data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=True, mlm_probability=0.15)
    trainer = Trainer(model=model, args=args, data_collator=data_collator,
                      train_dataset=train_dataset, eval_dataset=val_dataset, prediction_loss_only=True,)

    eval_loss = trainer.evaluate()
    eval_loss = eval_loss['eval_loss']
    logger.info(f'Initial eval bpc: {eval_loss/math.log(2)}')
    
    if not eval_only:
        trainer.train(model_path=model_path)
        trainer.save_model()

        eval_loss = trainer.evaluate()
        eval_loss = eval_loss['eval_loss']
        logger.info(f'Eval bpc after pretraining: {eval_loss/math.log(2)}')
```

With the [bert-base-multilingual-cased](https://storage.googleapis.com/bert_models/2018_11_23/multi_cased_L-12_H-768_A-12.zip) it works well: `model` and `tokenizer` passed as arguments to the function are respectively:

```python
model = BertForMaskedLM.from_pretrained('bert-base-multilingual-cased')
tokenizer = BertTokenizerFast.from_pretrained('bert-base-multilingual-cased')
```

But with the modified version of BERT this error occours:

```
Traceback (most recent call last):
  File "convert_bert_to_long_bert.py", line 172, in <module>
    pretrain_and_evaluate(training_args, model, tokenizer, eval_only=False, model_path=training_args.output_dir)
  File "convert_bert_to_long_bert.py", line 86, in pretrain_and_evaluate
    eval_loss = trainer.evaluate()
  File "/Users/user/Library/Python/3.7/lib/python/site-packages/transformers/trainer.py", line 748, in evaluate
    output = self._prediction_loop(eval_dataloader, description="Evaluation")
  File "/Users/user/Library/Python/3.7/lib/python/site-packages/transformers/trainer.py", line 829, in _prediction_loop
    outputs = model(**inputs)
  File "/Users/user/Library/Python/3.7/lib/python/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/user/Library/Python/3.7/lib/python/site-packages/transformers/modeling_bert.py", line 1098, in forward
    return_tuple=return_tuple,
  File "/Users/user/Library/Python/3.7/lib/python/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/user/Library/Python/3.7/lib/python/site-packages/transformers/modeling_bert.py", line 799, in forward
    return_tuple=return_tuple,
  File "/Users/user/Library/Python/3.7/lib/python/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/user/Library/Python/3.7/lib/python/site-packages/transformers/modeling_bert.py", line 460, in forward
    output_attentions,
  File "/Users/user/Library/Python/3.7/lib/python/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/user/Library/Python/3.7/lib/python/site-packages/transformers/modeling_bert.py", line 391, in forward
    hidden_states, attention_mask, head_mask, output_attentions=output_attentions,
  File "/Users/user/Library/Python/3.7/lib/python/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
  File "/Users/user/Library/Python/3.7/lib/python/site-packages/transformers/modeling_bert.py", line 335, in forward
    hidden_states, attention_mask, head_mask, encoder_hidden_states, encoder_attention_mask, output_attentions,
  File "/Users/user/Library/Python/3.7/lib/python/site-packages/torch/nn/modules/module.py", line 550, in __call__
    result = self.forward(*input, **kwargs)
TypeError: forward() takes from 2 to 4 positional arguments but 7 were given
```
I did few modifications to a working script to obtain a Long version of RoBERTa given the RoBERTa base model. What could be the mistake?

Comments:
**Update:** I have downgraded transformers to the version `transformers==2.11.0` and it seems working, even if for now I have used little datasets for test. I will update this issue if someone is interested

The code in Longformer has changed quite a bit. I think a simply remedy to make your code work with the current version of `Longformer` is to add `**kwargs` to every forward function in `modeling_longformer.py` that you copied into your notebook. This way it can handle an arbitrary number of input arguments and the above error should not occur.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.


> The code in Longformer has changed quite a bit. I think a simply remedy to make your code work with the current version of `Longformer` is to add `**kwargs` to every forward function in `modeling_longformer.py` that you copied into your notebook. This way it can handle an arbitrary number of input arguments and the above error should not occur.

EDIT: To begin pre-training, make sure you LOAD the saved model exactly the way the notebook does BEFORE pre-training! Don't try and use the model straightaway!

I have same issue and the problem remains. It looks the problem comes from a higher transformer version.