TensorFlow Question-Answering example fails to run (cardinality error)
Author: sfvaroglu
Pull request: no

## Environment info

- `transformers` version: 4.4.0.dev0
- Platform: Linux-4.15.0-111-generic-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.8
- PyTorch version (GPU?): 1.7.1 (False)
- Tensorflow version (GPU?): 2.2.0 (False)
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No

## Information

Model I am using (Bert, XLNet ...): bert-base-uncased or roberta-base

The problem arises when using:
* [ ] the official example scripts: question-answering (run_tf_squad.py)

Error message:
Instructions for updating:
back_prop=False is deprecated. Consider using tf.stop_gradient instead.
Instead of:
results = tf.map_fn(fn, elems, back_prop=False)
Use:
results = tf.nest.map_structure(tf.stop_gradient, tf.map_fn(fn, elems))
87599it [01:00, 1437.03it/s]
10570it [00:11, 958.83it/s]
convert squad examples to features:   2%|_         | 1697/87599 [00:13<10:43, 133.40it/s][WARNING|squad.py:118] 2021-02-17 22:20:03,736 >> Could not find answer: 'municipal building and' vs. 'a municipal building'
convert squad examples to features:  50%|_____     | 43393/87599 [05:04<05:04, 145.24it/s][WARNING|squad.py:118] 2021-02-17 22:24:55,103 >> Could not find answer: 'message stick,' vs. 'a message stick'
convert squad examples to features: 100%|__________| 87599/87599 [10:10<00:00, 143.59it/s]
add example index and unique id: 100%|__________| 87599/87599 [00:00<00:00, 784165.53it/s]
convert squad examples to features: 100%|__________| 10570/10570 [01:14<00:00, 140.99it/s]
add example index and unique id: 100%|__________| 10570/10570 [00:00<00:00, 510000.04it/s]
[WARNING|integrations.py:60] 2021-02-17 22:31:16,214 >> Using the `WAND_DISABLED` environment variable is deprecated and will be removed in v5. Use the --report_to flag to control the integrations used for logging result (for instance --report_to none).
[INFO|trainer_tf.py:125] 2021-02-17 22:31:16,214 >> To use comet_ml logging, run `pip/conda install comet_ml` see https://www.comet.ml/docs/python-sdk/huggingface/
Traceback (most recent call last):
  File "run_tf_squad.py", line 256, in <module>
    main()
  File "run_tf_squad.py", line 250, in main
    trainer.train()
  File "/home/transformers/src/transformers/trainer_tf.py", line 457, in train
    train_ds = self.get_train_tfdataset()
  File "/home/transformers/src/transformers/trainer_tf.py", line 141, in get_train_tfdataset
    self.num_train_examples = self.train_dataset.cardinality().numpy()
AttributeError: '_AssertCardinalityDataset' object has no attribute 'cardinality'

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: SQUaD v1

## To reproduce

1. Use the latest master from huggingface/transformers
2. Go to examples/question-answering
3. Run WANDB_DISABLED=true python run_tf_squad.py --model_name_or_path roberta-base --output_dir model --max_seq_length 384 --num_train_epochs 2 --per_device_train_batch_size 8 --per_device_eval_batch_size 16 --do_train --do_eval --logging_dir logs --logging_steps 10 --learning_rate 3e-5 --no_cuda=True --doc_stride 128

Could you take a look @sgugger?

Comments:
cc @jplu since it seems to come from the `TFTrainer`.

Hello!

Since transformers 4.2.0 you need to have TensorFlow 2.3 at least.

@jplu With TensorFlow 2.3 and transformers 4.4.0.dev0, I'm getting the error below:

[INFO|trainer_tf.py:522] 2021-02-18 19:18:25,103 >> ***** Running training *****
[INFO|trainer_tf.py:523] 2021-02-18 19:18:25,103 >>   Num examples = 87599
[INFO|trainer_tf.py:525] 2021-02-18 19:18:25,104 >>   Num Epochs = 2
[INFO|trainer_tf.py:526] 2021-02-18 19:18:25,104 >>   Instantaneous batch size per device = 8
[INFO|trainer_tf.py:528] 2021-02-18 19:18:25,105 >>   Total train batch size (w. parallel, distributed & accumulation) = 8
[INFO|trainer_tf.py:530] 2021-02-18 19:18:25,105 >>   Gradient Accumulation steps = 1
[INFO|trainer_tf.py:531] 2021-02-18 19:18:25,105 >>   Steps per epoch = 10950
[INFO|trainer_tf.py:532] 2021-02-18 19:18:25,105 >>   Total optimization steps = 21900
2021-02-18 19:18:25.182464: W tensorflow/core/framework/op_kernel.cc:1755] Invalid argument: TypeError: `generator` yielded an element that did not match the expected structure. The expected structure was ({'input_ids': tf.int32, 'attention_mask': tf.int32, 'feature_index': tf.int64, 'qas_id': tf.string}, {'
start_positions': tf.int64, 'end_positions': tf.int64, 'cls_index': tf.int64, 'p_mask': tf.int32, 'is_impossible': tf.int32}), but the yielded element was ({'input_ids': [0, 6179, 793, 21, 2708, 77, 79, 21, 39504, 8358, 25, 10, 33799, 116, 2, 2, 767, 7, 5, 6256, 45756, 34527, 24292, 9, 957, 6, 2708, 21, 5, 1
354, 9, 6130, 3889, 1488, 757, 8, 6130, 7896, 4, 3224, 2708, 18, 33694, 6, 7896, 56, 57, 36175, 8, 21, 444, 3319, 11, 107, 4, 2708, 21, 576, 7, 544, 25, 10, 39504, 8358, 33799, 11, 5, 9660, 11, 7007, 77, 79, 21, 130, 107, 793, 6, 203, 101, 11029, 362, 9581, 7, 5, 12765, 3281, 24618, 25, 2673, 11, 5, 3470, 36
209, 4, 993, 6256, 45756, 34527, 2349, 194, 14, 23, 5, 86, 9, 69, 5673, 1001, 27534, 7, 3351, 6, 2708, 21, 316, 2383, 1570, 107, 793, 6, 8, 37, 21, 16984, 107, 793, 6, 53, 215, 2349, 32, 31298, 4, 2, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
, 1, 1, 1, 1, 1, 1, 1], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'feature_index': 0, 'qas_id': '570c2b046b8089140040fba5'}, {'start_positions': 73, 'end_positions': 76,
'cls_index': 0, 'p_mask': [0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], 'is_impossible': False}).
Traceback (most recent call last):

  File "/home/svaroglu/.hf_venv/lib/python3.6/site-packages/tensorflow/python/data/ops/dataset_ops.py", line 833, in generator_py_func
    flattened_values = nest.flatten_up_to(output_types, values)

  File "/home/svaroglu/.hf_venv/lib/python3.6/site-packages/tensorflow/python/data/util/nest.py", line 396, in flatten_up_to
    assert_shallow_structure(shallow_tree, input_tree)

  File "/home/svaroglu/.hf_venv/lib/python3.6/site-packages/tensorflow/python/data/util/nest.py", line 324, in assert_shallow_structure
    check_types=check_types)

  File "/home/svaroglu/.hf_venv/lib/python3.6/site-packages/tensorflow/python/data/util/nest.py", line 311, in assert_shallow_structure
    % (len(input_tree), len(shallow_tree)))

ValueError: The two structures don't have the same sequence length. Input structure has length 5, while shallow structure has length 4.


This is because there is an issue in https://github.com/huggingface/transformers/blob/master/src/transformers/data/processors/squad.py .

We will look into it asap and will let you know here once done. Sorry for the inconvenience.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.

Hello, I am getting the same error. 

It says "AttributeError: 'Dataset' object has no attribute 'cardinality'" when I train it. Does anyone know how I should address this issue? 

> Hello, I am getting the same error.
> 
> It says "AttributeError: 'Dataset' object has no attribute 'cardinality'" when I train it. Does anyone know how I should address this issue?

@jeehunkang Could you find a solution to this?

 "AttributeError: 'Dataset' object has no attribute 'cardinality'" when I train with TFTrainer. my tf version is 2.14.
File Name: trainer_tf.py
Line #479     train_ds = self.get_train_tfdataset()
Line #159     self.num_train_examples = self.train_dataset.cardinality().numpy() in 

My Code snippet
training_args = TFTrainingArguments(
    output_dir=output_dir,
    learning_rate=1e-5,
    num_train_epochs=1,
    weight_decay=0.01,
    logging_steps=1,
    max_steps=1
)

trainer = TFTrainer(
    model=original_model,
    args=training_args,
    train_dataset=tokenized_datasets['train'],
    eval_dataset=tokenized_datasets['validation']
)

trainer.train()

Please note that TFTrainer is no longer actively maintained cc @Rocketknight1 

Yes, this is quite an old issue that dates back to a time before I wrote proper Keras support for our TF classes! If people are still getting errors running the modern `run_qa.py` example script or the question answering notebook, please open a new issue and paste your code so I can reproduce and diagnose the error!