The pytorch example question-answering/run_qa_beam_search.py do not work
Author: nicolalandro
Pull request: no

## Environment info

- `transformers` version: git+https://github.com/huggingface/transformers
- Platform:
- Python version: 3.8
- PyTorch version (GPU?): 1.10.0
- Using GPU in script?: yes

### Who can help

@pvl  @vanpelt  @NielsRogge @sgugger
 

Models:

- T5: gsarti/it5-base
- encoder-decoder models (For example, BlenderBot, BART, Marian, Pegasus, T5, ByT5): gsarti/it5-base
- Pytorch: 1.10.0

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

HF projects:

- datasets: [squad-it](https://huggingface.co/datasets/z-uo/squad-it), adapted from [github squad-it](https://github.com/crux82/squad-it)

Examples:

- maintained examples (not research project or legacy): [question-answering/run_qa_beam_search.py](https://github.com/huggingface/transformers/blob/master/examples/pytorch/question-answering/run_qa_beam_search.py)

## Information

Model I am using (Bert, XLNet ...): T5

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. clone code: `git clone https://gitlab.com/nicolalandro/qandatrain.git` (in this repo I copy the official files for train and I fix the lib in requirements)
2. go into the code folder: `cd qandatrain`
3. install requirements: `pip install -r requirements.txt`
4. clone dataset: `git clone https://huggingface.co/datasets/z-uo/squad-it`
5. run the code:
```
python src/run_qa_beam_search.py \
  --model_name_or_path gsarti/it5-base \
  --tokenizer_name gsarti/it5-base \
  --dataset_name squad \
  --train_file "squad-it/SQuAD_it-train_processed.json" \
  --validation_file "squad-it/SQuAD_it-test_processed.json" \
  --do_train \
  --do_eval \
  --per_device_train_batch_size 3 \
  --learning_rate 3e-5 \
  --num_train_epochs 2 \
  --max_seq_length 384 \
  --doc_stride 128 \
  --output_dir it5-squad
```
6. you obtain the following error:
```
...
Traceback (most recent call last):
  File "src/run_qa_beam_search.py", line 696, in <module>
    main()
  File "src/run_qa_beam_search.py", line 454, in main
    train_dataset = train_dataset.map(
  File "/media/mint/Barracuda/Project/qandatrain/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2036, in map
    return self._map_single(
  File "/media/mint/Barracuda/Project/qandatrain/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 503, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/media/mint/Barracuda/Project/qandatrain/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 470, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/media/mint/Barracuda/Project/qandatrain/venv/lib/python3.8/site-packages/datasets/fingerprint.py", line 406, in wrapper
    out = func(self, *args, **kwargs)
  File "/media/mint/Barracuda/Project/qandatrain/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2404, in _map_single
    batch = apply_function_on_filtered_inputs(
  File "/media/mint/Barracuda/Project/qandatrain/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2291, in apply_function_on_filtered_inputs
    function(*fn_args, effective_indices, **fn_kwargs) if with_indices else function(*fn_args, **fn_kwargs)
  File "/media/mint/Barracuda/Project/qandatrain/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1991, in decorated
    result = f(decorated_item, *args, **kwargs)
  File "src/run_qa_beam_search.py", line 386, in prepare_train_features
    cls_index = input_ids.index(tokenizer.cls_token_id)
ValueError: 32005 is not in list
```

It seams an error on the tokenizer that do not find some token on the dictionary or into the sentences.

## Expected behavior

Train the T5 model for question answering on squad-it and create the trained model files at output_dir


Comments:
> @arfon @pvl @vanpelt @karthikrangasai

Unsure why I received a ping here. Any clues people of Hugging Face?

> Unsure why I received a ping here. Any clues people of Hugging Face?
Github suggest you to me, If you cansusggest someone else, I can  change. 


@sgugger and @NielsRogge should be able to help.

Hi,

The `run_qa_beam_search.py` and `run_qa_beam_search_no_trainer.py` scripts are only meant for XLNet (which is a special, encoder-only model).

T5 is an encoder-decoder (seq2seq) model, which solves question-answering in a different way, as it's a generative model, rather than a discriminative one. It learns to generate the correct answer from a question + context, instead of predicting the position of the start and end token of the answer. 

You can take a look at the new `run_seq2seq_qa.py` [script](https://github.com/huggingface/transformers/blob/master/examples/pytorch/question-answering/run_seq2seq_qa.py) which was added 2 days ago.

Thank you for the explanation! the train now is started with the correct script, I will put the results on Hugging face if it will be good.

During the eval I have this error:
```
Traceback (most recent call last):
  File "src/run_seq2seq_qa.py", line 638, in <module>
    main()
  File "src/run_seq2seq_qa.py", line 597, in main
    metrics = trainer.evaluate(max_length=max_length, num_beams=num_beams, metric_key_prefix="eval")
  File "/media/mint/Barracuda/Project/qandatrain/venv/lib/python3.8/site-packages/transformers/trainer_seq2seq.py", line 75, in evaluate
    return super().evaluate(eval_dataset, ignore_keys=ignore_keys, metric_key_prefix=metric_key_prefix)
  File "/media/mint/Barracuda/Project/qandatrain/venv/lib/python3.8/site-packages/transformers/trainer.py", line 2113, in evaluate
    output = eval_loop(
  File "/media/mint/Barracuda/Project/qandatrain/venv/lib/python3.8/site-packages/transformers/trainer.py", line 2354, in evaluation_loop
    metrics = self.compute_metrics(EvalPrediction(predictions=all_preds, label_ids=all_labels))
  File "src/run_seq2seq_qa.py", line 539, in compute_metrics
    decoded_preds = [tokenizer.batch_decode(pred, skip_special_tokens=True) for pred in preds]
  File "src/run_seq2seq_qa.py", line 539, in <listcomp>
    decoded_preds = [tokenizer.batch_decode(pred, skip_special_tokens=True) for pred in preds]
  File "/media/mint/Barracuda/Project/qandatrain/venv/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 3182, in batch_decode
    return [
  File "/media/mint/Barracuda/Project/qandatrain/venv/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 3183, in <listcomp>
    self.decode(
  File "/media/mint/Barracuda/Project/qandatrain/venv/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 3221, in decode
    return self._decode(
  File "/media/mint/Barracuda/Project/qandatrain/venv/lib/python3.8/site-packages/transformers/tokenization_utils_fast.py", line 528, in _decode
    text = self._tokenizer.decode(token_ids, skip_special_tokens=skip_special_tokens)
TypeError: 'float' object cannot be interpreted as an integer
```

And if I do not limit the max_eval_sample=220 with 8GB of ram after some batch of eval it go in OOM maybe there is some memory leack.

@nicolalandro I had the same error when writing tests for the script. You should use the `--predict_with_generate` flag.

Perfect with that param The train ended correctly thank you!