[run_seq2seq_qa.py] various issues
Author: stas00
Pull request: no
Labels: name: bug

I was trying to use `examples/pytorch/question-answering/run_seq2seq_qa.py` to write a test and run into multiple issues.

A. the example is not working:

https://github.com/huggingface/transformers/blob/d1d5ebb16cc8500a3e4e1b30047312cc563ca87f/examples/pytorch/question-answering/README.md#fine-tuning-t5-on-squad20


1. running as is fails with:

```
ValueError: --answer_column' value 'answer' needs to be one of: id, title, context, question, answers
```

The example should say `--answer_column answers` (not `answer`)

2. ok, trying to move forward:

```
$ python examples/pytorch/question-answering/run_seq2seq_qa.py   --model_name_or_path t5-small   --dataset_name squad_v2   --context_column context   --question_column question   --answer_column answers   --do_train   --do_eval   --per_device_train_batch_size 12   --learning_rate 3e-5   --num_train_epochs 1   --max_seq_length 384   --doc_stride 128   --output_dir /tmp/debug_seq2seq_squad/
```
crashes with:
```
05/11/2022 18:02:23 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/stas/.cache/huggingface/datasets/squad_v2/squad_v2/2.0.0/09187c73c1b837c95d9a249cd97c2c3f1cebada06efe667b4427714b27639b1d/cache-d9a027917b78cfa7.arrow
Running tokenizer on validation dataset:   0%|                                                                     | 0/12 [00:00<?, ?ba/s]05/11/2022 18:02:23 - INFO - datasets.arrow_dataset - Caching processed dataset at /home/stas/.cache/huggingface/datasets/squad_v2/squad_v2/2.0.0/09187c73c1b837c95d9a249cd97c2c3f1cebada06efe667b4427714b27639b1d/cache-0b463497dc4250c6.arrow
Running tokenizer on validation dataset:   0%|                                                                     | 0/12 [00:03<?, ?ba/s]
Traceback (most recent call last):
  File "examples/pytorch/question-answering/run_seq2seq_qa.py", line 687, in <module>
    main()
  File "examples/pytorch/question-answering/run_seq2seq_qa.py", line 519, in main
    eval_dataset = eval_examples.map(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2346, in map
    return self._map_single(
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 532, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 499, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/fingerprint.py", line 458, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2751, in _map_single
    writer.write_batch(batch)
  File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/arrow_writer.py", line 510, in write_batch
    pa_table = pa.Table.from_arrays(arrays, schema=schema)
  File "pyarrow/table.pxi", line 1702, in pyarrow.lib.Table.from_arrays
  File "pyarrow/table.pxi", line 1314, in pyarrow.lib.Table.validate
  File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 4 named labels expected length 1007 but got length 1000
```

B. If I try to switch from `run_qa.py` to this script, e.g.:

```
python examples/pytorch/question-answering/run_seq2seq_qa.py --model_name_or_path valhalla/t5-base-squad --tokenizer_name valhalla/t5-base-squad --dataset_name squad --output_dir ./xxx --overwrite_output_dir --optim adafactor --do_train --max_train_samples 3 --do_eval --max_eval_samples 1 --logging_strategy steps --logging_steps 1 --evaluation_strategy steps --eval_steps 1 --save_strategy steps --save_steps 1 --load_best_model_at_end --per_device_train_batch_size 1 --per_device_eval_batch_size 1 --num_train_epochs 1 --report_to none --fp16
```
it crashes on:

```
Traceback (most recent call last):                                                                                  | 0/1 [00:00<?, ?it/s]
  File "examples/pytorch/question-answering/run_seq2seq_qa.py", line 687, in <module>
    main()
  File "examples/pytorch/question-answering/run_seq2seq_qa.py", line 623, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/trainer.py", line 1317, in train
    return inner_training_loop(
  File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/trainer.py", line 1629, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/trainer.py", line 1801, in _maybe_log_save_evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
  File "/mnt/nvme0/code/huggingface/transformers-master/examples/pytorch/question-answering/trainer_seq2seq_qa.py", line 71, in evaluate
    eval_preds = self.post_process_function(eval_examples, eval_dataset, output)
  File "examples/pytorch/question-answering/run_seq2seq_qa.py", line 580, in post_processing_function
    decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)
  File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 3287, in batch_decode
    return [
  File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 3288, in <listcomp>
    self.decode(
  File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 3326, in decode
    return self._decode(
  File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/tokenization_utils_fast.py", line 547, in _decode
    text = self._tokenizer.decode(token_ids, skip_special_tokens=skip_special_tokens)
TypeError: 'list' object cannot be interpreted as an integer
```

basically:

```
preds == [[[nan, nan, ..., nan]]]
```
and there are 2 problems here:
1. it has one level too many of nesting - hence the error above
2. if I manually tweak it to pass `preds[0]` it then fails to deal with `nan` and then fails with:

```
Traceback (most recent call last):
  File "examples/pytorch/question-answering/run_seq2seq_qa.py", line 687, in <module>
    main()
  File "examples/pytorch/question-answering/run_seq2seq_qa.py", line 623, in main
    train_result = trainer.train(resume_from_checkpoint=checkpoint)
  File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/trainer.py", line 1317, in train
    return inner_training_loop(
  File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/trainer.py", line 1629, in _inner_training_loop
    self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
  File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/trainer.py", line 1801, in _maybe_log_save_evaluate
    metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
  File "/mnt/nvme0/code/huggingface/transformers-master/examples/pytorch/question-answering/trainer_seq2seq_qa.py", line 71, in evaluate
    eval_preds = self.post_process_function(eval_examples, eval_dataset, output)
  File "examples/pytorch/question-answering/run_seq2seq_qa.py", line 580, in post_processing_function
    decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)
  File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 3287, in batch_decode
    return [
  File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 3288, in <listcomp>
    self.decode(
  File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 3326, in decode
    return self._decode(
  File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/tokenization_utils_fast.py", line 548, in _decode
    text = self._tokenizer.decode(token_ids[0], skip_special_tokens=skip_special_tokens)
TypeError: 'float' object cannot be interpreted as an integer
```

C. The test that exercises this script uses a local sample and it succeeds:

```
python examples/pytorch/question-answering/run_seq2seq_qa.py \
            --model_name_or_path t5-small \
            --context_column context \
            --question_column question \
            --answer_column answers \
            --version_2_with_negative \
            --train_file tests/fixtures/tests_samples/SQUAD/sample.json \
            --validation_file tests/fixtures/tests_samples/SQUAD/sample.json \
            --output_dir /tmp/debug_seq2seq_squad/ \
            --overwrite_output_dir \
            --max_steps=10 \
            --warmup_steps=2 \
            --do_train \
            --do_eval \
            --learning_rate=2e-4 \
            --per_device_train_batch_size=2 \
            --per_device_eval_batch_size=1 \
            --predict_with_generate
```

but once switching to `--dataset_name squad_v2` it breaks.

Environment-wise I'm using all the latest versions of datasets, transformers, etc. Please let me know if you need any specific versions of anything if you can't reproduce those issues.

### Who can help?

Tagging @karthikrangasai who created this script, but of course if others know how to fix it please don't hesitate to step in. Thank you!


Comments:
No answer from @karthikrangasai, @LysandreJik - who would be a good person to look at these issues? Thank you.

I believe @patil-suraj has experience with similar scripts, would you like to have a look at this one when you have a minute, @patil-suraj ?

Great! Thank you for tagging Suraj, Lysandre! and thank you, Suraj for checking it

Is there any update on this?

I am also trying to use the script. Additionally, I have found some more issues with this:

1. It uses the doc_stride strategy to break long contexts, but at the end of the evaluation, no special handling is done and it seems it just takes into account the latest feature extracted from an example (which seems to be not a good approach)
2.  The post_process script is based on the feature set having an `example_id` column, but the Trainer hides that column, and the script breaks in that part. In the provided Colab, there is a tweak that "resets" the features dataset format for it to work. Maybe bringing this to this script?


 I hope this helps. It would be great to have a script for that ....

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.