Getting KeyError: 'loss' when fine-tuning model on a pre-trained MLM
Author: neel04
Pull request: no

## Environment info
- `transformers` version: 4.5.1
- Platform: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.10
- PyTorch version (GPU?): 1.8.1+cu101 (False)
- Tensorflow version (GPU?): 2.4.1 (False)
- Using GPU in script?: Depends - CPU for debugging
- Using distributed or parallel set-up in script?: False

### Who can help
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten

- tokenizers: @LysandreJik
## Information

Model I am using (Bert, XLNet ...): Longformer (custom upload on Model Hub)

The problem arises when using:
* [ ] the official example scripts: (give details below)
* [x] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [x] my own task or dataset: (give details below)

## To reproduce

I created a LM model and uploaded it to Huggingface Model Hub via [this colab notebook](https://colab.research.google.com/drive/153754DbFXRhKdHvjdSUUp9VSB5JqtZwX?usp=sharing)

But when fine-tuning the model on simple reproducible data, I get:-

```py
%%bash
pip install -q transformers
pip install -q datasets

import numpy as np
train_text = np.array(['a foxy', 'b ball', 'c cats r bad', 'as das', 'sagha','asdfsd','asd','ad','aets','hsdg','reya','arey','areyareh','yui','aEWY','DSH','ASUYH','ASFH','ASDFHG','OOO'], dtype='<U5280')
train_label = [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20]
val_text = np.array(['a foxy', 'r c cats'], dtype='<U5280')
val_label = [1, 2]

from datasets import Dataset
train_dataset = Dataset.from_dict({'src':train_text, 'tgt':train_label})
val_dataset = Dataset.from_dict({'src':val_text, 'tgt':val_label})

from transformers import AutoTokenizer
    
tokenizer = AutoTokenizer.from_pretrained("MalawiUniST/ISO6392.nya.ny", use_fast=True, truncation=True, padding=True, max_length=10) #try fast=False
def tok(example):
  encodings = tokenizer(example['src'], truncation=True, padding="max_length", max_length=10)
  return encodings

train_encoded_dataset = train_dataset.map(tok, batched=True)
val_encoded_dataset = val_dataset.map(tok, batched=True)

from transformers import Trainer, TrainingArguments, AutoModelForSequenceClassification

from sklearn.metrics import accuracy_score, precision_recall_fscore_support

def compute_metrics(pred):
    labels = pred.label_ids
    preds = pred.predictions.argmax(-1)
    precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='weighted',zero_division=1)  #none gives score for each class
    acc = accuracy_score(labels, preds)
    return {
        'accuracy': acc,
        'f1': f1,
        'precision': precision,
        'recall': recall
    }

training_args = TrainingArguments(
    output_dir='/content/results/',          # output directory
    overwrite_output_dir = True,
    num_train_epochs=16,              # total number of training epochs
    per_device_train_batch_size=32,  # batch size per device during training
    per_device_eval_batch_size=32,   # batch size for evaluation
    warmup_steps=600,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='/content/logs',            # directory for storing logs
    logging_steps=10,
    evaluation_strategy='epoch',
    learning_rate=1e-6,
    #fp16 = True,
    load_best_model_at_end = True,
    metric_for_best_model = 'eval_loss',
    greater_is_better = False,
    seed = 101,
    save_total_limit=5,
)

model = AutoModelForSequenceClassification.from_pretrained("MalawiUniST/ISO6392.nya.ny", num_labels=20)

trainer = Trainer(
    model=model,                         # the instantiated Transformers model to be trained
    args=training_args,                  # training arguments, defined above
    train_dataset=train_encoded_dataset,         # training dataset
    eval_dataset=val_encoded_dataset,             # evaluation dataset
    compute_metrics=compute_metrics,
    tokenizer=tokenizer
    )

train_results = trainer.train()
```

This error:-

```
Some weights of the model checkpoint at MalawiUniST/ISO6392.nya.ny were not used when initializing LongformerForSequenceClassification: ['lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight', 'lm_head.decoder.bias']
- This IS expected if you are initializing LongformerForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing LongformerForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of LongformerForSequenceClassification were not initialized from the model checkpoint at MalawiUniST/ISO6392.nya.ny and are newly initialized: ['classifier.dense.weight', 'classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.out_proj.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

---------------------------------------------------------------------------

KeyError                                  Traceback (most recent call last)

<ipython-input-113-a2ff149dfd3d> in <module>()
     46     )
     47 
---> 48 train_results = trainer.train()

3 frames

/usr/local/lib/python3.7/dist-packages/transformers/trainer.py in train(self, resume_from_checkpoint, trial, **kwargs)
   1118                         tr_loss += self.training_step(model, inputs)
   1119                 else:
-> 1120                     tr_loss += self.training_step(model, inputs)
   1121                 self._total_flos += float(self.floating_point_ops(inputs))
   1122 

/usr/local/lib/python3.7/dist-packages/transformers/trainer.py in training_step(self, model, inputs)
   1522                 loss = self.compute_loss(model, inputs)
   1523         else:
-> 1524             loss = self.compute_loss(model, inputs)
   1525 
   1526         if self.args.n_gpu > 1:

/usr/local/lib/python3.7/dist-packages/transformers/trainer.py in compute_loss(self, model, inputs, return_outputs)
   1564         else:
   1565             # We don't use .loss here since the model may return tuples instead of ModelOutput.
-> 1566             loss = outputs["loss"] if isinstance(outputs, dict) else outputs[0]
   1567 
   1568         return (loss, outputs) if return_outputs else loss

/usr/local/lib/python3.7/dist-packages/transformers/file_utils.py in __getitem__(self, k)
   1614         if isinstance(k, str):
   1615             inner_dict = {k: v for (k, v) in self.items()}
-> 1616             return inner_dict[k]
   1617         else:
   1618             return self.to_tuple()[k]

KeyError: 'loss'
```

From sgugger's reply [here on forums](https://discuss.huggingface.co/t/key-error-loss-while-fine-tuning-gpt-2-with-the-trainer-utility/2861/4?u=neel-gupta) it seems that one strong cause is when the labels aren't present (even though they certainly are upon printing it out)

This seems like a bug and the code is reproducible on colab. Any ideas to possible workarounds?

Comments:
Hi @neel04. Regarding your code sample with simple reproducible data. I believe there are two errors here:
- First, you've created your labels as a range from 1 to 20. You've set the model's `num_labels` to 20, but that, unfortunately, means it has a range of `[0, 19]`, therefore unable to satisfy the label 20. I would start the labels at 0.
- Secondly, and more importantly, in your encoding method you're only tokenizing the source input, and doing nothing to your target input:

```py
def tok(example):
  encodings = tokenizer(example['src'], truncation=True, padding="max_length", max_length=10)
  return encodings

train_encoded_dataset = train_dataset.map(tok, batched=True)
val_encoded_dataset = val_dataset.map(tok, batched=True)
```

Therefore, here, your `train_encoded_dataset` and `val_encoded_dataset` contain dictionaries with the following keys: `input_ids` and `attention_mask`. There are no labels.

You could manually add your labels to your encoding by tweaking your `tok` method:

```py
def tok(example):
  encodings = tokenizer(example['src'], truncation=True, padding="max_length", max_length=10)
  encodings["labels"] = example["tgt"]
  return encodings

train_encoded_dataset = train_dataset.map(tok, batched=True)
val_encoded_dataset = val_dataset.map(tok, batched=True)
```

Otherwise, instead of naming your variable `tgt` inside the dataset, you could name it `labels` so that it's adequately named in your `Dataset` right away.

I've been running your colab, but didn't run into an issue yet, I'm at step 2000. If I run into an issue, I'll try to see what's going on.

Hope that helps.

Thanx a ton for replying @LysandreJik !!! :hugs: :+1: 
About the labels, I think you may be right - I totally missed that point.

secondly, are you not seeing `tgt` in the `train_encoded_dataset` inside the repro? I do see it when printing it out :thinking: 

> I've been running your colab

do you mean you are re-training the LM? I already have it on model hub BTW - can you fine-tune that pre-trained model successfully? 

I'm sorry, you are correct, the `dataset` has the following attributes: `['attention_mask', 'input_ids', 'src', 'tgt']`. However, the model only cares about the `attention_mask` and `input_ids`. It also cares about the `labels`, which are absent in this case, hence why your code was failing.

If you want to have a look at what inputs the model needs, I encourage you to take a look at the docs; you're using `LongformerForSequenceClassification`, see the parameters it acepts [here](https://huggingface.co/transformers/model_doc/longformer.html#transformers.LongformerForSequenceClassification.forward).

I did manage to run your code example, but I thought the colab would fail in a similar fashion. It seems it trained correctly, so that is not an issue.

Is there anything else you need help with?

Thanx for the help! I am surprised why we need to add a `labels` attribute since we specify it when constructing the `Dataset` object - so it must be easy for HF to guess the numerical value and use it as labels accordingly. 

it does work for repro, so the issue does not remain now - but I would greatly appreciate if you can help me out! I am trying to train it on my normal data.
I have used the `train_text_split` -er to split into NumPy arrays and am trying to pass it - but it still gives me the index error.

Repro:
```py
Dataset({
    features: ['attention_mask', 'input_ids', 'labels', 'src', 'tgt'],
    num_rows: 20
})
```
Main dataset:
```py
Dataset({
    features: ['attention_mask', 'input_ids', 'labels', 'src', 'tgt'],
    num_rows: 4572
})
```
There doesn't seem any surface difference; I checked the length of the mask and ids - they are as expected. checked 'labels', is numeric - doesn't cross 20.

Clearly, there is some problem with my input data. Could you give an idea about what the error might indicate is wrong with my data?
My input data is basically documents - long strings. they are cleaned thoroughly and are purely text. the only thing is that they are pretty long (sometimes longer than 2000 tokens). 
Any opinions on what the issue could be?

> I am surprised why we need to add a labels attribute since we specify it when constructing the Dataset object - so it must be easy for HF to guess the numerical value and use it as labels accordingly.

Are you referencing the fact that we're passing the `Dataset` a `tgt` value during initialization? If so, then yes those are labels but since the model looks for the field `labels`, it will not look at `tgt`. If you define it as `labels` right off the bat, it should work!

Regarding your second question, I fear I'm out of the loop. If you have an example which fail, that would be great, with a minimal reproducible code example, that would be even better!

Do you have an idea of which document especially might cause an issue? Thank you.

I was able to repro the issue with this dummy dataset:
```py
import numpy as np
train_text = np.array(['lorem ipsum'*499]*20)
train_label = [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
val_text = np.array(['lorem ipsum'*499]*2)
val_label = [1, 2]
```
What's your take on it? looks like a bug for high length strings - Even though they do seem padded and truncated via `datasets`. 

---

**EDIT:-** This is the full error, in case you want something to refer to, instead of running your own code
```py

Downloading: 100%
199M/199M [00:09<00:00, 21.2MB/s]


Some weights of the model checkpoint at MalawiUniST/ISO6392.nya.ny were not used when initializing LongformerForSequenceClassification: ['lm_head.bias', 'lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.weight', 'lm_head.layer_norm.bias', 'lm_head.decoder.weight', 'lm_head.decoder.bias']
- This IS expected if you are initializing LongformerForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing LongformerForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of LongformerForSequenceClassification were not initialized from the model checkpoint at MalawiUniST/ISO6392.nya.ny and are newly initialized: ['classifier.dense.weight', 'classifier.dense.bias', 'classifier.out_proj.weight', 'classifier.out_proj.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

---------------------------------------------------------------------------

IndexError                                Traceback (most recent call last)

<ipython-input-9-d3bd01a1a0a7> in <module>()
     46     )
     47 
---> 48 train_results = trainer.train()

11 frames

/usr/local/lib/python3.7/dist-packages/transformers/trainer.py in train(self, resume_from_checkpoint, trial, **kwargs)
   1118                         tr_loss += self.training_step(model, inputs)
   1119                 else:
-> 1120                     tr_loss += self.training_step(model, inputs)
   1121                 self._total_flos += float(self.floating_point_ops(inputs))
   1122 

/usr/local/lib/python3.7/dist-packages/transformers/trainer.py in training_step(self, model, inputs)
   1522                 loss = self.compute_loss(model, inputs)
   1523         else:
-> 1524             loss = self.compute_loss(model, inputs)
   1525 
   1526         if self.args.n_gpu > 1:

/usr/local/lib/python3.7/dist-packages/transformers/trainer.py in compute_loss(self, model, inputs, return_outputs)
   1554         else:
   1555             labels = None
-> 1556         outputs = model(**inputs)
   1557         # Save past state if it exists
   1558         # TODO: this needs to be fixed and made cleaner later.

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/usr/local/lib/python3.7/dist-packages/transformers/models/longformer/modeling_longformer.py in forward(self, input_ids, attention_mask, global_attention_mask, head_mask, token_type_ids, position_ids, inputs_embeds, labels, output_attentions, output_hidden_states, return_dict)
   1855             output_attentions=output_attentions,
   1856             output_hidden_states=output_hidden_states,
-> 1857             return_dict=return_dict,
   1858         )
   1859         sequence_output = outputs[0]

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/usr/local/lib/python3.7/dist-packages/transformers/models/longformer/modeling_longformer.py in forward(self, input_ids, attention_mask, global_attention_mask, head_mask, token_type_ids, position_ids, inputs_embeds, output_attentions, output_hidden_states, return_dict)
   1662 
   1663         embedding_output = self.embeddings(
-> 1664             input_ids=input_ids, position_ids=position_ids, token_type_ids=token_type_ids, inputs_embeds=inputs_embeds
   1665         )
   1666 

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/usr/local/lib/python3.7/dist-packages/transformers/models/longformer/modeling_longformer.py in forward(self, input_ids, token_type_ids, position_ids, inputs_embeds)
    491         if inputs_embeds is None:
    492             inputs_embeds = self.word_embeddings(input_ids)
--> 493         position_embeddings = self.position_embeddings(position_ids)
    494         token_type_embeddings = self.token_type_embeddings(token_type_ids)
    495 

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
    887             result = self._slow_forward(*input, **kwargs)
    888         else:
--> 889             result = self.forward(*input, **kwargs)
    890         for hook in itertools.chain(
    891                 _global_forward_hooks.values(),

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/sparse.py in forward(self, input)
    156         return F.embedding(
    157             input, self.weight, self.padding_idx, self.max_norm,
--> 158             self.norm_type, self.scale_grad_by_freq, self.sparse)
    159 
    160     def extra_repr(self) -> str:

/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in embedding(input, weight, padding_idx, max_norm, norm_type, scale_grad_by_freq, sparse)
   1914         # remove once script supports set_grad_enabled
   1915         _no_grad_embedding_renorm_(weight, input, max_norm, norm_type)
-> 1916     return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
   1917 
   1918 

IndexError: index out of range in self


```

Thank you for the clear reproducible example and the error stack trace.

That's curious, it does not break on my machine; running the following code, which is a concatenation of the sample you've just given me regarding the dataset and the training code of your initial issue description:

```py
import numpy as np
train_text = np.array(['lorem ipsum'*499]*20)
train_label = [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
val_text = np.array(['lorem ipsum'*499]*2)
val_label = [1, 2]

from datasets import Dataset

train_dataset = Dataset.from_dict({'src': train_text, 'tgt': train_label})
val_dataset = Dataset.from_dict({'src': val_text, 'tgt': val_label})

from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("MalawiUniST/ISO6392.nya.ny", use_fast=True, truncation=True, padding=True,
                                          max_length=10)  # try fast=False


def tok(example):
    encodings = tokenizer(example['src'], truncation=True, padding="max_length", max_length=10)
    encodings["labels"] = example["tgt"]
    return encodings


train_encoded_dataset = train_dataset.map(tok, batched=True)
val_encoded_dataset = val_dataset.map(tok, batched=True)

print(train_encoded_dataset)

from transformers import Trainer, TrainingArguments, AutoModelForSequenceClassification

from sklearn.metrics import accuracy_score, precision_recall_fscore_support

def compute_metrics(pred):
    labels = pred.label_ids
    preds = pred.predictions.argmax(-1)
    precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='weighted',zero_division=1)  #none gives score for each class
    acc = accuracy_score(labels, preds)
    return {
        'accuracy': acc,
        'f1': f1,
        'precision': precision,
        'recall': recall
    }

training_args = TrainingArguments(
    output_dir='content/results/',          # output directory
    overwrite_output_dir = True,
    num_train_epochs=16,              # total number of training epochs
    per_device_train_batch_size=32,  # batch size per device during training
    per_device_eval_batch_size=32,   # batch size for evaluation
    warmup_steps=600,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='content/logs',            # directory for storing logs
    logging_steps=10,
    evaluation_strategy='epoch',
    learning_rate=1e-6,
    #fp16 = True,
    load_best_model_at_end = True,
    metric_for_best_model = 'eval_loss',
    greater_is_better = False,
    seed = 101,
    save_total_limit=5,
)

model = AutoModelForSequenceClassification.from_pretrained("MalawiUniST/ISO6392.nya.ny", num_labels=20)

trainer = Trainer(
    model=model,                         # the instantiated Transformers model to be trained
    args=training_args,                  # training arguments, defined above
    train_dataset=train_encoded_dataset,         # training dataset
    eval_dataset=val_encoded_dataset,             # evaluation dataset
    compute_metrics=compute_metrics,
    tokenizer=tokenizer
    )

train_results = trainer.train()
```

However, in your error, I understand that there's an `IndexError` happening with the position embeddings, the interesting line being this one:
```
--> 493         position_embeddings = self.position_embeddings(position_ids)
```
This is very frequently an issue with padding/truncation, as you have correctly identified. I can indeed reproduce if I remove the notion of padding/truncation from your tokenizer call in your `tok` method:

```diff
def tok(example):
-   encodings = tokenizer(example['src'], truncation=True, padding="max_length", max_length=10)
+   encodings = tokenizer(example['src'])
    encodings["labels"] = example["tgt"]
    return encodings
```

I think it is my fault :sweat_smile:  I had changed `max_length=10` to ` max_length=2000` which is the appropriate length it was intended for and pre-trained on. Maybe that's why it ran on your machine, but failed on Colab?

About the padding/truncation indeed, I am using the way it's marked in red - and can confirm that the length of each `attention mask`  is 2000, along with the `input_ids`. since that's the case for samples, the only conclusion is that I am indeed padding and truncating the sequences.

So the last point for the error is the `max_length` - which can't be 2000. in the LM (accessible via the colab link in OP, fully reproducible end-to-end example) the tokenizer construction is like this:-
```py
model_checkpoint = "allenai/longformer-base-4096"
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True, max_length=2000)
```
which does specify 2000 to be the maximum length. I can probably try re-training the model, but it doesn't make sense If I can't find why (and how) the error originates and what changes to make. 

Any suspicions?


With a little trial and error, I got `max_length=500` to be the maximum I can use - losing out a lot of information :face_with_head_bandage:  This seems like a very weird bug. Everything seems normal, but its always errors after crossing 500

Is it failing after crossing 500 or 512? It's possible that there's a rogue max length of 512 (which obviously shouldn't be here!)

Surprisingly I'm having issues reproducing the error with your maximum length of 2000, which doesn't crash on my side either (as it shouldn't!)

Do you have an example I can run locally which fails with length > 500?

yep, it def gets an error in Colab with CPU. for making sure repro changes, I will put the whole thing in here (with the dummy data):-
```py
import numpy as np
train_text = np.array(['lorem ipsum'*499]*20)
train_label = [0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19]
val_text = np.array(['lorem ipsum'*499]*2)
val_label = [1, 2]

MAX_LENGTH = 2000

!pip install -q transformers
!pip install -q datasets
import transformers
transformers.__version__

from datasets import Dataset
train_dataset = Dataset.from_dict({'src':train_text, 'tgt':train_label})
val_dataset = Dataset.from_dict({'src':val_text, 'tgt':val_label})
from transformers import AutoTokenizer
    
tokenizer = AutoTokenizer.from_pretrained("MalawiUniST/ISO6392.nya.ny", use_fast=True, truncation=True, padding=True, max_length=MAX_LENGTH) #try fast=False

def tok(example):
  encodings = tokenizer(example['src'], truncation=True, padding=True, max_length=MAX_LENGTH)
  encodings["labels"] = example["tgt"]                         #Try removing this line
  return encodings

len(train_encoded_dataset['attention_mask'][0])

from transformers import Trainer, TrainingArguments, AutoModelForSequenceClassification

from sklearn.metrics import accuracy_score, precision_recall_fscore_support

def compute_metrics(pred):
    labels = pred.label_ids
    preds = pred.predictions.argmax(-1)
    precision, recall, f1, _ = precision_recall_fscore_support(labels, preds, average='weighted',zero_division=1)  #none gives score for each class
    acc = accuracy_score(labels, preds)
    return {
        'accuracy': acc,
        'f1': f1,
        'precision': precision,
        'recall': recall
    }

training_args = TrainingArguments(
    output_dir='/content/results/',          # output directory
    overwrite_output_dir = True,
    num_train_epochs=16,              # total number of training epochs
    per_device_train_batch_size=32,  # batch size per device during training
    per_device_eval_batch_size=32,   # batch size for evaluation
    warmup_steps=600,                # number of warmup steps for learning rate scheduler
    weight_decay=0.01,               # strength of weight decay
    logging_dir='/content/logs',            # directory for storing logs
    logging_steps=10,
    evaluation_strategy='epoch',
    learning_rate=1e-6,
    #fp16 = True,
    load_best_model_at_end = True,
    metric_for_best_model = 'eval_loss',
    greater_is_better = False,
    seed = 101,
    save_total_limit=5,
)

model = AutoModelForSequenceClassification.from_pretrained("MalawiUniST/ISO6392.nya.ny", num_labels=20)

trainer = Trainer(
    model=model,                         # the instantiated Transformers model to be trained
    args=training_args,                  # training arguments, defined above
    train_dataset=train_encoded_dataset,         # training dataset
    eval_dataset=val_encoded_dataset,             # evaluation dataset
    compute_metrics=compute_metrics,
    tokenizer=tokenizer
    )

train_results = trainer.train()
```
should reproduce the error on pasting straightaway! :+1: 

---

**EDIT:-** yep, you are right - using `max_length` as `513` gets an error, and 512 doesn't. I am using Longformer here - so the whole situation becomes tricky :thinking: By default, `Longformer-base-4096` should get `4096` as the max_length.

When pre-training the LM, this is the snippet for initializing the model from scratch:
```py
from transformers import LongformerForMaskedLM
from transformers import LongformerConfig

config = LongformerConfig(
    vocab_size=52_000,
    max_position_embeddings=514,
    num_attention_heads=2,
    num_hidden_layers=1,
    type_vocab_size=1,
)

model = LongformerForMaskedLM(config=config)
```
tokenizer too is gotten properly:
```py
model_checkpoint = "allenai/longformer-base-4096"
tokenizer = AutoTokenizer.from_pretrained(model_checkpoint, use_fast=True, max_length=2000)
```
Very strange.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.