text: 
  
  TITLE
  [run_seq2seq_qa.py] various issues
  
  COMMENTS
  0
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  I was trying to use `examples/pytorch/question-answering/run_seq2seq_qa.py` to write a test and run into multiple issues.
  
  A. the example is not working:
  
  https://github.com/huggingface/transformers/blob/d1d5ebb16cc8500a3e4e1b30047312cc563ca87f/examples/pytorch/question-answering/README.md#fine-tuning-t5-on-squad20
  
  
  1. running as is fails with:
  
  ```
  ValueError: --answer_column' value 'answer' needs to be one of: id, title, context, question, answers
  ```
  
  The example should say `--answer_column answers` (not `answer`)
  
  2. ok, trying to move forward:
  
  ```
  $ python examples/pytorch/question-answering/run_seq2seq_qa.py   --model_name_or_path t5-small   --dataset_name squad_v2   --context_column context   --question_column question   --answer_column answers   --do_train   --do_eval   --per_device_train_batch_size 12   --learning_rate 3e-5   --num_train_epochs 1   --max_seq_length 384   --doc_stride 128   --output_dir /tmp/debug_seq2seq_squad/
  ```
  crashes with:
  ```
  05/11/2022 18:02:23 - WARNING - datasets.arrow_dataset - Loading cached processed dataset at /home/stas/.cache/huggingface/datasets/squad_v2/squad_v2/2.0.0/09187c73c1b837c95d9a249cd97c2c3f1cebada06efe667b4427714b27639b1d/cache-d9a027917b78cfa7.arrow
  Running tokenizer on validation dataset:   0%|                                                                     | 0/12 [00:00<?, ?ba/s]05/11/2022 18:02:23 - INFO - datasets.arrow_dataset - Caching processed dataset at /home/stas/.cache/huggingface/datasets/squad_v2/squad_v2/2.0.0/09187c73c1b837c95d9a249cd97c2c3f1cebada06efe667b4427714b27639b1d/cache-0b463497dc4250c6.arrow
  Running tokenizer on validation dataset:   0%|                                                                     | 0/12 [00:03<?, ?ba/s]
  Traceback (most recent call last):
    File "examples/pytorch/question-answering/run_seq2seq_qa.py", line 687, in <module>
      main()
    File "examples/pytorch/question-answering/run_seq2seq_qa.py", line 519, in main
      eval_dataset = eval_examples.map(
    File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2346, in map
      return self._map_single(
    File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 532, in wrapper
      out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 499, in wrapper
      out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/fingerprint.py", line 458, in wrapper
      out = func(self, *args, **kwargs)
    File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2751, in _map_single
      writer.write_batch(batch)
    File "/home/stas/anaconda3/envs/py38-pt111/lib/python3.8/site-packages/datasets/arrow_writer.py", line 510, in write_batch
      pa_table = pa.Table.from_arrays(arrays, schema=schema)
    File "pyarrow/table.pxi", line 1702, in pyarrow.lib.Table.from_arrays
    File "pyarrow/table.pxi", line 1314, in pyarrow.lib.Table.validate
    File "pyarrow/error.pxi", line 99, in pyarrow.lib.check_status
  pyarrow.lib.ArrowInvalid: Column 4 named labels expected length 1007 but got length 1000
  ```
  
  B. If I try to switch from `run_qa.py` to this script, e.g.:
  
  ```
  python examples/pytorch/question-answering/run_seq2seq_qa.py --model_name_or_path valhalla/t5-base-squad --tokenizer_name valhalla/t5-base-squad --dataset_name squad --output_dir ./xxx --overwrite_output_dir --optim adafactor --do_train --max_train_samples 3 --do_eval --max_eval_samples 1 --logging_strategy steps --logging_steps 1 --evaluation_strategy steps --eval_steps 1 --save_strategy steps --save_steps 1 --load_best_model_at_end --per_device_train_batch_size 1 --per_device_eval_batch_size 1 --num_train_epochs 1 --report_to none --fp16
  ```
  it crashes on:
  
  ```
  Traceback (most recent call last):                                                                                  | 0/1 [00:00<?, ?it/s]
    File "examples/pytorch/question-answering/run_seq2seq_qa.py", line 687, in <module>
      main()
    File "examples/pytorch/question-answering/run_seq2seq_qa.py", line 623, in main
      train_result = trainer.train(resume_from_checkpoint=checkpoint)
    File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/trainer.py", line 1317, in train
      return inner_training_loop(
    File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/trainer.py", line 1629, in _inner_training_loop
      self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
    File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/trainer.py", line 1801, in _maybe_log_save_evaluate
      metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
    File "/mnt/nvme0/code/huggingface/transformers-master/examples/pytorch/question-answering/trainer_seq2seq_qa.py", line 71, in evaluate
      eval_preds = self.post_process_function(eval_examples, eval_dataset, output)
    File "examples/pytorch/question-answering/run_seq2seq_qa.py", line 580, in post_processing_function
      decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)
    File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 3287, in batch_decode
      return [
    File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 3288, in <listcomp>
      self.decode(
    File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 3326, in decode
      return self._decode(
    File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/tokenization_utils_fast.py", line 547, in _decode
      text = self._tokenizer.decode(token_ids, skip_special_tokens=skip_special_tokens)
  TypeError: 'list' object cannot be interpreted as an integer
  ```
  
  basically:
  
  ```
  preds == [[[nan, nan, ..., nan]]]
  ```
  and there are 2 problems here:
  1. it has one level too many of nesting - hence the error above
  2. if I manually tweak it to pass `preds[0]` it then fails to deal with `nan` and then fails with:
  
  ```
  Traceback (most recent call last):
    File "examples/pytorch/question-answering/run_seq2seq_qa.py", line 687, in <module>
      main()
    File "examples/pytorch/question-answering/run_seq2seq_qa.py", line 623, in main
      train_result = trainer.train(resume_from_checkpoint=checkpoint)
    File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/trainer.py", line 1317, in train
      return inner_training_loop(
    File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/trainer.py", line 1629, in _inner_training_loop
      self._maybe_log_save_evaluate(tr_loss, model, trial, epoch, ignore_keys_for_eval)
    File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/trainer.py", line 1801, in _maybe_log_save_evaluate
      metrics = self.evaluate(ignore_keys=ignore_keys_for_eval)
    File "/mnt/nvme0/code/huggingface/transformers-master/examples/pytorch/question-answering/trainer_seq2seq_qa.py", line 71, in evaluate
      eval_preds = self.post_process_function(eval_examples, eval_dataset, output)
    File "examples/pytorch/question-answering/run_seq2seq_qa.py", line 580, in post_processing_function
      decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)
    File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 3287, in batch_decode
      return [
    File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 3288, in <listcomp>
      self.decode(
    File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/tokenization_utils_base.py", line 3326, in decode
      return self._decode(
    File "/mnt/nvme0/code/huggingface/transformers-master/src/transformers/tokenization_utils_fast.py", line 548, in _decode
      text = self._tokenizer.decode(token_ids[0], skip_special_tokens=skip_special_tokens)
  TypeError: 'float' object cannot be interpreted as an integer
  ```
  
  C. The test that exercises this script uses a local sample and it succeeds:
  
  ```
  python examples/pytorch/question-answering/run_seq2seq_qa.py \
              --model_name_or_path t5-small \
              --context_column context \
              --question_column question \
              --answer_column answers \
              --version_2_with_negative \
              --train_file tests/fixtures/tests_samples/SQUAD/sample.json \
              --validation_file tests/fixtures/tests_samples/SQUAD/sample.json \
              --output_dir /tmp/debug_seq2seq_squad/ \
              --overwrite_output_dir \
              --max_steps=10 \
              --warmup_steps=2 \
              --do_train \
              --do_eval \
              --learning_rate=2e-4 \
              --per_device_train_batch_size=2 \
              --per_device_eval_batch_size=1 \
              --predict_with_generate
  ```
  
  but once switching to `--dataset_name squad_v2` it breaks.
  
  Environment-wise I'm using all the latest versions of datasets, transformers, etc. Please let me know if you need any specific versions of anything if you can't reproduce those issues.
  
  ### Who can help?
  
  Tagging @karthikrangasai who created this script, but of course if others know how to fix it please don't hesitate to step in. Thank you!
  