ValueError("The training dataset must have an asserted cardinality") when running run_tf_ner.py
Author: Xuanfang1121
Pull request: no
Labels: name: wontfix

## Environment info
- `transformers` version: 4.2.0
- Platform: linux
- Python version: python3.6
- PyTorch version (GPU?): 1.7.1 gpu
- Tensorflow version (GPU?):2.4.0
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?:no

### Who can help
 @stefan-it
 
## Information

Model I am using (Bert, XLNet ...): bert-base-multilingual-cased

The problem arises when using:

- [yes ] the official example scripts: (give details below)

* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name) ner  GermEval 2014 
* [ ] my own task or dataset: (give details below)

## To reproduce

Steps to reproduce the behavior:

1. Created a new conda environment using conda env -n xftf2 python=3.6
2. pip insrall transformer==4.2.0 tensorflow==2.4 torch==1.7.1
3. Prepare the data set(train.txt, test.txt, dev.txt) according to the README under the folder token-classification, run run_tf_ner.py 
   setting  from_pt=True, with the following parameters:
```
--data_dir ./data \
--labels ./data/labels.txt \
--model_name_or_path bert-base-multilingual-cased \
--output_dir ./output \
--max_seq_length  128 \
--num_train_epochs 4\
--per_device_train_batch_size 32 \
--save_steps 500 \
--seed 100 \
--do_train \
--do_eval \
--do_predict
```
Here is the stack trace:
```
01/21/2021 00:12:18 - INFO - utils_ner -   *** Example ***
01/21/2021 00:12:18 - INFO - utils_ner -   guid: dev-5
01/21/2021 00:12:18 - INFO - utils_ner -   tokens: [CLS] Dara ##us entwickelte sich im Rok ##oko die Sitt ##e des gemeinsamen Wein ##ens im Theater , das die Stand ##es ##grenze ##n innerhalb des Publikum ##s über ##brücken sollte . [SEP]
01/21/2021 00:12:18 - INFO - utils_ner -   input_ids: 101 95621 10251 28069 10372 10211 51588 20954 10128 105987 10112 10139 58090 90462 12457 10211 16223 117 10242 10128 15883 10171 58433 10115 21103 10139 63332 10107 10848 99765 17799 119 102 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/21/2021 00:12:18 - INFO - utils_ner -   input_mask: 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/21/2021 00:12:18 - INFO - utils_ner -   segment_ids: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
01/21/2021 00:12:18 - INFO - utils_ner -   label_ids: -1 24 -1 24 24 24 6 -1 24 24 -1 24 24 24 -1 24 24 24 24 24 24 -1 -1 -1 24 24 24 -1 24 -1 24 24 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1 -1
Traceback (most recent call last):
  File "run_tf_ner.py", line 299, in <module>
    main()
  File "run_tf_ner.py", line 231, in main
    trainer.train()
  File "/.conda/envs/xftf2/lib/python3.6/site-packages/transformers/trainer_tf.py", line 457, in train
    train_ds = self.get_train_tfdataset()
  File "/.conda/envs/xftf2/lib/python3.6/site-packages/transformers/trainer_tf.py", line 141, in get_train_tfdataset
    raise ValueError("The training dataset must have an asserted cardinality")
ValueError: The training dataset must have an asserted cardinality
```

## Expected behavior
In such a case, is there any tips to deal with it?I really appreciate any help you can provide.


Comments:
Maybe @jplu has an idea!

Hello!

This error is always raised by the TFTrainer when your dataset has not a cardinality attached.

Can you give me the version of the `run_tf_ner.py` you are using please?

The run_tf_ner.py I used was downloaded from this https://github.com/huggingface/transformers/tree/master/examples/token-classification， transformer version is 4.2.0， tensorflow == 2.4.0
 @jplu 

Are you sure this is the exact version or not from another commit? Because I see a cardinality assigned in the current script. Even thought the script is not working since 4.2.0 but for a diffeerent reason.

This issue has been automatically marked as stale and been closed because it has not had recent activity. Thank you for your contributions.

If you think this still needs to be addressed please comment on this thread.