Repository: huggingface/transformers

Issue:
ValueError("The training dataset must have an asserted cardinality") when running run_tf_text_classification.py
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.3.1 (installed from master)
- Platform: Linux-4.15.0-118-generic-x86_64-with-debian-stretch-sid
- Python version: 3.7.9
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): 2.3.1 (True)
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 
 albert, bert, GPT2, XLM: @LysandreJik 
 tokenizers: @mfuntowicz
 Trainer: @sgugger
 Speed and Memory Benchmarks: @patrickvonplaten
 Model Cards: @julien-c
 Translation: @sshleifer
 Summarization: @sshleifer
 TextGeneration: @TevenLeScao 
 examples/distillation: @VictorSanh
 nlp datasets: [different repo](https://github.com/huggingface/nlp)
 rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)
 Text Generation: @TevenLeScao
 blenderbot: @mariamabarham
 Bart: @sshleifer
 Marian: @sshleifer
 T5: @patrickvonplaten
 Longformer/Reformer: @patrickvonplaten
 TransfoXL/XLNet: @TevenLeScao 
 examples/seq2seq: @sshleifer
 examples/bert-loses-patience: @JetRunner
 tensorflow: @jplu
 examples/token-classification: @stefan-it
 documentation: @sgugger
 -->
@jplu 

## Information

Model I am using (Bert, XLNet ...): Bert (bert-base-uncased)

The problem arises when using:
* [x] the official example scripts: (give details below)
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [x] an official GLUE/SQUaD task: SST-2
* [x] my own task or dataset: (give details below)

This same problem happened to my custom dataset, as I described here in #7535 , and also using SST-2 from GLUE (which I did to confirm the error). The following steps are using SST-2 with bert-base-uncased.

## To reproduce

Steps to reproduce the behavior:

1. Created a new conda environment using conda env -n transformers python=3.7
2. Cloned transformers master, `cd` into it and installed using pip install --editable .  -r examples/requirements.txt 
3. Installed tensorflow with `pip install tensorflow`
4. Updated datasets to version 1.1.1, as needed according to issue #7535 
5. Ran `run_tf_text_classification.py` with the following parameters:

```
--train_file <DATASET_PATH>/train.csv \
--dev_file <DATASET_PATH>/dev.csv \ 
--test_file <DATASET_PATH>/dev.csv \
--label_column_id 1 \
--model_name_or_path bert-base-uncased \
--output_dir <OUTPUT_PATH> \
--num_train_epochs 4 \
--per_device_train_batch_size 4 \
--per_device_eval_batch_size 4 \
--do_train \
--do_eval \
--do_predict \
--logging_steps 1000 \
--evaluate_during_training \
--save_steps 1000 \
--overwrite_output_dir \
--overwrite_cache
```


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

Here is the stack trace:

```
10/07/2020 09:48:49 - INFO - __main__ -   Training/evaluation parameters TFTrainingArguments(output_dir='/media/discoD/models/datalawyer/pedidos/transformers_tf', overwrite_output_dir=True, do_train=True, do_eval=True, do_predict=True, evaluate_during_training=True, evaluation_strategy=<EvaluationStrategy.STEPS: 'steps'>, prediction_loss_only=False, per_device_train_batch_size=1, per_device_eval_batch_size=1, per_gpu_train_batch_size=None, per_gpu_eval_batch_size=None, gradient_accumulation_steps=1, learning_rate=5e-05, weight_decay=0.0, adam_beta1=0.9, adam_beta2=0.999, adam_epsilon=1e-08, max_grad_norm=1.0, num_train_epochs=4.0, max_steps=-1, warmup_steps=0, logging_dir='runs/Oct07_09-48-45_user-XPS-8700', logging_first_step=False, logging_steps=10000, save_steps=10000, save_total_limit=None, no_cuda=False, seed=42, fp16=False, fp16_opt_level='O1', local_rank=-1, tpu_num_cores=None, tpu_metrics_debug=False, debug=False, dataloader_drop_last=False, eval_steps=10000, dataloader_num_workers=0, past_index=-1, run_name='/media/discoD/models/datalawyer/pedidos/transformers_tf', disable_tqdm=False, remove_unused_columns=True, label_names=None, load_best_model_at_end=False, metric_for_best_model=None, greater_is_better=False, tpu_name=None, xla=False)
10/07/2020 09:48:52 - INFO - filelock -   Lock 140079222710992 acquired on /home/user/.cache/huggingface/datasets/c19c3494c195b40ef4234cb533a8f3ce0bca75ffcf602cc246c390073e633c46.1d5301eeb143a6a4f6f3a2bf726921db0de85048303426a3810f96d735d50d8a.py.lock
10/07/2020 09:48:52 - INFO - filelock -   Lock 140079222710992 released on /home/user/.cache/huggingface/datasets/c19c3494c195b40ef4234cb533a8f3ce0bca75ffcf602cc246c390073e633c46.1d5301eeb143a6a4f6f3a2bf726921db0de85048303426a3810f96d735d50d8a.py.lock
Using custom data configuration default
10/07/2020 09:48:52 - INFO - filelock -   Lock 140084305595600 acquired on /home/user/.cache/huggingface/datasets/_home_user_.cache_huggingface_datasets_csv_default-477ee137eed7e5ae_0.0.0_49187751790fa4d820300fd4d0707896e5b941f1a9c644652645b866716a4ac4.lock
10/07/2020 09:48:52 - INFO - filelock -   Lock 140084305595600 released on /home/user/.cache/huggingface/datasets/_home_user_.cache_huggingface_datasets_csv_default-477ee137eed7e5ae_0.0.0_49187751790fa4d820300fd4d0707896e5b941f1a9c644652645b866716a4ac4.lock
10/07/2020 09:48:52 - INFO - filelock -   Lock 140080785346896 acquired on /home/user/.cache/huggingface/datasets/_home_user_.cache_huggingface_datasets_csv_default-477ee137eed7e5ae_0.0.0_49187751790fa4d820300fd4d0707896e5b941f1a9c644652645b866716a4ac4.lock
Reusing dataset csv (/home/user/.cache/huggingface/datasets/csv/default-477ee137eed7e5ae/0.0.0/49187751790fa4d820300fd4d0707896e5b941f1a9c644652645b866716a4ac4)
10/07/2020 09:48:52 - INFO - filelock -   Lock 140080785346896 released on /home/user/.cache/huggingface/datasets/_home_user_.cache_huggingface_datasets_csv_default-477ee137eed7e5ae_0.0.0_49187751790fa4d820300fd4d0707896e5b941f1a9c644652645b866716a4ac4.lock
100%|██████████| 68/68 [01:20<00:00,  1.18s/ba]
100%|██████████| 1/1 [00:01<00:00,  1.71s/ba]
100%|██████████| 1/1 [00:01<00:00,  1.44s/ba]
10/07/2020 09:50:23 - INFO - filelock -   Lock 140078150630032 acquired on /home/user/.cache/torch/transformers/336363d3718f8cc6432db4a768a053f96a9eae064c8c96aff2bc69fe73929770.4733ec82e81d40e9cf5fd04556267d8958fb150e9339390fc64206b7e5a79c83.h5.lock
Downloading: 100%|██████████| 536M/536M [04:08<00:00, 2.16MB/s]
10/07/2020 09:54:32 - INFO - filelock -   Lock 140078150630032 released on /home/user/.cache/torch/transformers/336363d3718f8cc6432db4a768a053f96a9eae064c8c96aff2bc69fe73929770.4733ec82e81d40e9cf5fd04556267d8958fb150e9339390fc64206b7e5a79c83.h5.lock
2020-10-07 09:54:46.214922: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 93763584 exceeds 10% of free system memory.
Some weights of the model checkpoint at bert-base-uncased were not used when initializing TFBertForSequenceClassification: ['nsp___cls', 'mlm___cls']
- This IS expected if you are initializing TFBertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing TFBertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of TFBertForSequenceClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['dropout_37', 'classifier']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Traceback (most recent call last):
  File "/media/discoD/pycharm-community-2019.2/plugins/python-ce/helpers/pydev/pydevd.py", line 1448, in _exec
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "/media/discoD/pycharm-community-2019.2/plugins/python-ce/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "/media/discoD/repositorios/transformers_pedro/examples/text-classification/run_tf_text_classification.py", line 283, in <module>
    main()
  File "/media/discoD/repositorios/transformers_pedro/examples/text-classification/run_tf_text_classification.py", line 258, in main
    trainer.train()
  File "/media/discoD/repositorios/transformers_pedro/src/transformers/trainer_tf.py", line 474, in train
    train_ds = self.get_train_tfdataset()
  File "/media/discoD/repositorios/transformers_pedro/src/transformers/trainer_tf.py", line 140, in get_train_tfdataset
    raise ValueError("The training dataset must have an asserted cardinality")
ValueError: The training dataset must have an asserted cardinality
```

## Expected behavior

Should be able to run the text-classification example as described in [https://github.com/huggingface/transformers/tree/master/examples/text-classification#run-generic-text-classification-script-in-tensorflow](https://github.com/huggingface/transformers/tree/master/examples/text-classification#run-generic-text-classification-script-in-tensorflow)

## An additional info: For my own data, using our bert-portuguese model, we don't have a model based on tensorflow available. So I had to force `from_pt` in the code below to be True, otherwise I would get a different error. The [script which converts pytorch to tensorflow](https://github.com/huggingface/transformers/blob/master/src/transformers/convert_bert_pytorch_checkpoint_to_original_tf.py) doesn't work with TF 2.0. 

```
    with training_args.strategy.scope():
        model = TFAutoModelForSequenceClassification.from_pretrained(
            model_args.model_name_or_path,
            from_pt=bool(".bin" in model_args.model_name_or_path),
            config=config,
            cache_dir=model_args.cache_dir,
        )
```


Discussion:

Resolving patch:
diff --git a/examples/text-classification/run_tf_text_classification.py b/examples/text-classification/run_tf_text_classification.py
--- a/examples/text-classification/run_tf_text_classification.py
+++ b/examples/text-classification/run_tf_text_classification.py
@@ -96,6 +96,9 @@ def gen_test():
         else None
     )
 
+    if train_ds is not None:
+        train_ds = train_ds.apply(tf.data.experimental.assert_cardinality(len(ds[datasets.Split.TRAIN])))
+
     val_ds = (
         tf.data.Dataset.from_generator(
             gen_val,
@@ -106,6 +109,9 @@ def gen_test():
         else None
     )
 
+    if val_ds is not None:
+        val_ds = val_ds.apply(tf.data.experimental.assert_cardinality(len(ds[datasets.Split.VALIDATION])))
+
     test_ds = (
         tf.data.Dataset.from_generator(
             gen_test,
@@ -116,6 +122,9 @@ def gen_test():
         else None
     )
 
+    if test_ds is not None:
+        test_ds = test_ds.apply(tf.data.experimental.assert_cardinality(len(ds[datasets.Split.TEST])))
+
     return train_ds, val_ds, test_ds, label2id
 
 


Test patch: