KeyError: 'num_special_tokens_to_add'
Author: elcolie
Pull request: no

### System Info

transformers==4.28.1
M2 MBP
OSX 13.2
Python 3.10.10

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```python
from transformers import GPT2Tokenizer, GPT2LMHeadModel, Trainer, TrainingArguments, TextDataset

# Step 1: Load the pre-trained GPT-2 model
model = GPT2LMHeadModel.from_pretrained('gpt2')

# Step 2: Tokenize the training data
tokenizer = GPT2Tokenizer.from_pretrained('gpt2')
train_file_path = 'shakespeare.txt'
train_encodings = tokenizer(train_file_path)

# Step 3: Prepare the training data
train_dataset = TextDataset(train_encodings, file_path=train_file_path, block_size=512)

# Step 4: Create a TrainingArguments object
training_args = TrainingArguments(
    output_dir='./results',
    num_train_epochs=3,
    per_device_train_batch_size=2,
    per_device_eval_batch_size=2,
    warmup_steps=500,
    weight_decay=0.01,
    logging_dir='./logs',
    logging_steps=1000,
    save_steps=5000,
    evaluation_strategy='steps',
    eval_steps=5000,
    load_best_model_at_end=True
)

# Step 5: Instantiate a Trainer object
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=train_dataset
)

# Step 6: Train the model
trainer.train()

```

[shakespeare.txt](https://github.com/huggingface/transformers/files/11440814/shakespeare.txt)



### Expected behavior

Successful fine tune the model.

**As is:**<br>
```bash
  warnings.warn(
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/sarit/miniforge3/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:248 │
│ in __getattr__                                                                                   │
│                                                                                                  │
│    245 │                                                                                         │
│    246 │   def __getattr__(self, item: str):                                                     │
│    247 │   │   try:                                                                              │
│ ❱  248 │   │   │   return self.data[item]                                                        │
│    249 │   │   except KeyError:                                                                  │
│    250 │   │   │   raise AttributeError                                                          │
│    251                                                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
KeyError: 'num_special_tokens_to_add'

During handling of the above exception, another exception occurred:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/sarit/study/gpt4all/gpt2_fine_tune.py:12 in <module>                                      │
│                                                                                                  │
│    9 train_encodings = tokenizer(train_file_path)                                                │
│   10                                                                                             │
│   11 # Step 3: Prepare the training data                                                         │
│ ❱ 12 train_dataset = TextDataset(train_encodings, file_path=train_file_path, block_size=512)     │
│   13                                                                                             │
│   14 # Step 4: Create a TrainingArguments object                                                 │
│   15 training_args = TrainingArguments(                                                          │
│                                                                                                  │
│ /Users/sarit/miniforge3/lib/python3.10/site-packages/transformers/data/datasets/language_modelin │
│ g.py:62 in __init__                                                                              │
│                                                                                                  │
│    59 │   │   if os.path.isfile(file_path) is False:                                             │
│    60 │   │   │   raise ValueError(f"Input file path {file_path} not found")                     │
│    61 │   │                                                                                      │
│ ❱  62 │   │   block_size = block_size - tokenizer.num_special_tokens_to_add(pair=False)          │
│    63 │   │                                                                                      │
│    64 │   │   directory, filename = os.path.split(file_path)                                     │
│    65 │   │   cached_features_file = os.path.join(                                               │
│                                                                                                  │
│ /Users/sarit/miniforge3/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:250 │
│ in __getattr__                                                                                   │
│                                                                                                  │
│    247 │   │   try:                                                                              │
│    248 │   │   │   return self.data[item]                                                        │
│    249 │   │   except KeyError:                                                                  │
│ ❱  250 │   │   │   raise AttributeError                                                          │
│    251 │                                                                                         │
│    252 │   def __getstate__(self):                                                               │
│    253 │   │   return {"data": self.data, "encodings": self._encodings}                          │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
```

Comments:
Hi @elcolie, 

The error is arising because `TextDataset` takes a `tokenizer` object as its first argument for instantiation. `train_encodings` is a dictionary containing the input ids and attention mask for the text input "shakespeare.txt" to be fed to  the model. This is what you want: 

```
train_dataset = TextDataset(tokenizer, file_path=train_file_path, block_size=512)
```

Note, TextDataset is deprecated and will soon be removed from the library. Preprocessing of datasets should be handled with the 🤗 Datasets library. You can see examples of how to use it in our [example scripts](https://github.com/huggingface/transformers/tree/main/examples) e.g. [this one for language modeling](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_mlm.py).

@amyeroberts 
I got new error. Thank you :)

```bash
Traceback (most recent call last):
  File "/Users/sarit/study/gpt4all/gpt2_fine_tune.py", line 37, in <module>
    trainer.train()
  File "/Users/sarit/miniforge3/lib/python3.10/site-packages/transformers/trainer.py", line 1662, in train
    return inner_training_loop(
  File "/Users/sarit/miniforge3/lib/python3.10/site-packages/transformers/trainer.py", line 1929, in _inner_training_loop
    tr_loss_step = self.training_step(model, inputs)
  File "/Users/sarit/miniforge3/lib/python3.10/site-packages/transformers/trainer.py", line 2692, in training_step
    inputs = self._prepare_inputs(inputs)
  File "/Users/sarit/miniforge3/lib/python3.10/site-packages/transformers/trainer.py", line 2639, in _prepare_inputs
    raise ValueError(
ValueError: The batch received was empty, your model won't be able to train on it. Double-check that your training dataset contains keys expected by the model: input_ids,past_key_values,attention_mask,token_type_ids,position_ids,head_mask,inputs_embeds,encoder_hidden_states,encoder_attention_mask,labels,use_cache,output_attentions,output_hidden_states,return_dict,label,label_ids,labels.
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /Users/sarit/study/gpt4all/gpt2_fine_tune.py:37 in <module>                                      │
│                                                                                                  │
│   34 )                                                                                           │
│   35                                                                                             │
│   36 # Step 6: Train the model                                                                   │
│ ❱ 37 trainer.train()                                                                             │
│   38                                                                                             │
│                                                                                                  │
│ /Users/sarit/miniforge3/lib/python3.10/site-packages/transformers/trainer.py:1662 in train       │
│                                                                                                  │
│   1659 │   │   inner_training_loop = find_executable_batch_size(                                 │
│   1660 │   │   │   self._inner_training_loop, self._train_batch_size, args.auto_find_batch_size  │
│   1661 │   │   )                                                                                 │
│ ❱ 1662 │   │   return inner_training_loop(                                                       │
│   1663 │   │   │   args=args,                                                                    │
│   1664 │   │   │   resume_from_checkpoint=resume_from_checkpoint,                                │
│   1665 │   │   │   trial=trial,                                                                  │
│                                                                                                  │
│ /Users/sarit/miniforge3/lib/python3.10/site-packages/transformers/trainer.py:1929 in             │
│ _inner_training_loop                                                                             │
│                                                                                                  │
│   1926 │   │   │   │   │   with model.no_sync():                                                 │
│   1927 │   │   │   │   │   │   tr_loss_step = self.training_step(model, inputs)                  │
│   1928 │   │   │   │   else:                                                                     │
│ ❱ 1929 │   │   │   │   │   tr_loss_step = self.training_step(model, inputs)                      │
│   1930 │   │   │   │                                                                             │
│   1931 │   │   │   │   if (                                                                      │
│   1932 │   │   │   │   │   args.logging_nan_inf_filter                                           │
│                                                                                                  │
│ /Users/sarit/miniforge3/lib/python3.10/site-packages/transformers/trainer.py:2692 in             │
│ training_step                                                                                    │
│                                                                                                  │
│   2689 │   │   │   `torch.Tensor`: The tensor with training loss on this batch.                  │
│   2690 │   │   """                                                                               │
│   2691 │   │   model.train()                                                                     │
│ ❱ 2692 │   │   inputs = self._prepare_inputs(inputs)                                             │
│   2693 │   │                                                                                     │
│   2694 │   │   if is_sagemaker_mp_enabled():                                                     │
│   2695 │   │   │   loss_mb = smp_forward_backward(model, inputs, self.args.gradient_accumulatio  │
│                                                                                                  │
│ /Users/sarit/miniforge3/lib/python3.10/site-packages/transformers/trainer.py:2639 in             │
│ _prepare_inputs                                                                                  │
│                                                                                                  │
│   2636 │   │   """                                                                               │
│   2637 │   │   inputs = self._prepare_input(inputs)                                              │
│   2638 │   │   if len(inputs) == 0:                                                              │
│ ❱ 2639 │   │   │   raise ValueError(                                                             │
│   2640 │   │   │   │   "The batch received was empty, your model won't be able to train on it.   │
│   2641 │   │   │   │   f"training dataset contains keys expected by the model: {','.join(self._  │
│   2642 │   │   │   )                                                                             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
ValueError: The batch received was empty, your model won't be able to train on it. Double-check that your training dataset contains keys expected by the model:
input_ids,past_key_values,attention_mask,token_type_ids,position_ids,head_mask,inputs_embeds,encoder_hidden_states,encoder_attention_mask,labels,use_cache,output_attentions,output_hidden_st
ates,return_dict,label,label_ids,labels.
```


@elcolie can you please tell me how you solve this new error?

`ValueError: The batch received was empty, your model won't be able to train on it. Double-check that your training dataset contains keys expected by the model:
input_ids,past_key_values,attention_mask,token_type_ids,position_ids,head_mask,inputs_embeds,encoder_hidden_states,encoder_attention_mask,labels,use_cache,output_attentions,output_hidden_st
ates,return_dict,label,label_ids,labels.`

@osman-haider Sorry, I forgot.