When trying to train LongformerModel got **forward() got an unexpected keyword argument 'labels'**
Author: MutuLawyer
Pull request: no
Labels: name: wontfix

@sgugger 
@patrickvonplaten 

## Environment info
- `transformers` version: 3.1.0
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0+cu101 (True)
- Tensorflow version (GPU?): not installed (NA)
- Using GPU in script?: True
- Using distributed or parallel set-up in script?: False
     
The model I am using: Longformer

The problem arises when using my own modified scripts (based on [How to train a new language model from scratch using Transformers and Tokenizers](https://github.com/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb) )

```
!pip uninstall -y tensorflow
!pip install git+https://github.com/huggingface/transformers
!pip list | grep -E 'transformers|tokenizers'
from pathlib import Path
from tokenizers import (ByteLevelBPETokenizer,
                             CharBPETokenizer,
                             SentencePieceBPETokenizer,
                             BertWordPieceTokenizer)
tokenizer = ByteLevelBPETokenizer()
 
tokenizer.train(files="/content/drive/My Drive/summa/news_text.txt", vocab_size=5000, min_frequency=5, special_tokens=[
     "
",
     "<pad>",
     "
",
     "<unk>",
     "<mask>",
 ])
!mkdir LongformerL
tokenizer.save_model("./LongformerL")
from tokenizers.implementations import CharBPETokenizer
from tokenizers.processors import BertProcessing
tokenizer = ByteLevelBPETokenizer(
    "./LongformerL/vocab.json",
    "./LongformerL/merges.txt",
)
tokenizer._tokenizer.post_processor = BertProcessing(
    ("
", tokenizer.token_to_id("
")),
    ("
", tokenizer.token_to_id("
")),
)
tokenizer.enable_truncation(max_length=512)
from transformers import LongformerConfig, LongformerModel
config = LongformerConfig(
    vocab_size=5_000,
    max_position_embeddings=514,
    num_attention_heads=12,
    num_hidden_layers=6,
    type_vocab_size=1,
)
from transformers import LongformerTokenizerFast
tokenizer = LongformerTokenizerFast.from_pretrained("./LongformerL", max_len=512)
from transformers import LongformerModel
model = LongformerModel(config=config)
model.num_parameters()
from transformers import LineByLineTextDataset
dataset = LineByLineTextDataset(
     tokenizer=tokenizer,
     file_path="/content/drive/My Drive/summa/news_text.txt",
     block_size=128,
 )
from transformers import DataCollatorForLanguageModeling
data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer, mlm=True, mlm_probability=0.15
)
from transformers import Trainer, TrainingArguments
training_args = TrainingArguments(
    output_dir="/content/drive/My Drive/dataset_contract",
    overwrite_output_dir=True,
    per_device_train_batch_size=2,
    num_train_epochs=5,
    save_steps=1_000,
    save_total_limit=2,
)
trainer = Trainer(
    model=model,
    args=training_args,
    data_collator=data_collator,
    train_dataset=dataset,
    prediction_loss_only=True,
)
trainer.train()

```
The tasks I am working on is my own task or dataset. I'd like to train a Longformer from scratch. 

```
Epoch: 0%
0/5 [00:00<?, ?it/s]
Iteration: 0%
0/50 [00:00<?, ?it/s]
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-47-0c647bc3a8b8> in <module>()
----> 1 get_ipython().run_cell_magic('time', '', 'trainer.train()')

6 frames
<decorator-gen-60> in time(self, line, cell, local_ns)

<timed eval> in <module>()

/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    720             result = self._slow_forward(*input, **kwargs)
    721         else:
--> 722             result = self.forward(*input, **kwargs)
    723         for hook in itertools.chain(
    724                 _global_forward_hooks.values(),

**TypeError: forward() got an unexpected keyword argument 'labels'**
```


Comments:
The base models for all architectures are not usable with `Trainer` since they don't have a training objective. You need to pick the model class adapted to your task.

> The base models for all architectures are not usable with `Trainer` since they don't have a training objective. You need to pick the model class adapted to your task.

Thank you!
I replaced the model class "LongformerModel" with the "LongformerForSequenceClassification".
and got an error. 
ValueError: Expected input batch_size (2) to match target batch_size (256).
Which class of Longformer model is suitable for me (if any)?
Actually I'd like to perform a multi-classification task with my dataset.

```
ValueError                                Traceback (most recent call last)
<ipython-input-76-0c647bc3a8b8> in <module>()
----> 1 get_ipython().run_cell_magic('time', '', 'trainer.train()')

11 frames
<decorator-gen-60> in time(self, line, cell, local_ns)

<timed eval> in <module>()

/usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction)
   2214     if input.size(0) != target.size(0):
   2215         raise ValueError('Expected input batch_size ({}) to match target batch_size ({}).'
-> 2216                          .format(input.size(0), target.size(0)))
   2217     if dim == 2:
   2218         ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index)

ValueError: Expected input batch_size (2) to match target batch_size (256).
```

You can't use a `DataCollatorForLanguageModeling` for a classification task, this data collator prepares the data for language modeling (with masking since you set that option to `True`).

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.