cannot determine what will be the cardinality of the output after applying glue_convert_examples_to_features [TF  2.2.0rcx]
Author: tarrade
Pull request: no

# 🐛 Bug

## Information
With Tensorflow 2.2.0 (2.2.0rc2) we should be able to see the number of entries in the data without looking over them and using tf.data.experimental.cardinality. 

One issue that I found is that after applying  `glue_convert_examples_to_features` tf.data.experimental.cardinality is not able to find the total number of entry. I thought first that it was bug in this TF 2.2.0 release candidate.https://github.com/tensorflow/tensorflow/issues/37998. 

When using data from tensorflow dataset tf.data.experimental.cardinality is returning the number of event

```
print(data['train'])
print(tf.data.experimental.cardinality(data['train'])) 
```

```
<DatasetV1Adapter shapes: {idx: (), label: (), sentence: ()}, types: {idx: tf.int32, label: tf.int64, sentence: tf.string}>
tf.Tensor(67349, shape=(), dtype=int64)
```

Now when I am using Huggingface transformer that modify the structure of the data:
```
train_dataset = glue_convert_examples_to_features(data['train'], 
                                                  tokenizer, 
                                                  max_length=128, 
                                                  task='sst-2')

print(tf.data.experimental.cardinality(train_dataset))
```

```
<FlatMapDataset shapes: ({input_ids: (None,), attention_mask: (None,), token_type_ids: (None,)}, ()), types: ({input_ids: tf.int32, attention_mask: tf.int32, token_type_ids: tf.int32}, tf.int64)>
tf.Tensor(-2, shape=(), dtype=int64)
```

When the input pipeline contains a flat_map, it is generally not possible to statically determine what will be the cardinality of the output from the cardinality from the input. I don't see any flatmap in this function. I am trying to identify which part of the code is responsible. I am not 100% sure this is a transformer issue.

## To reproduce

Steps to reproduce the behavior:

```
import tensorflow as tf
import tensorflow_datasets

from transformers import (
    BertConfig,
    BertTokenizer,
    TFBertModel,
    TFBertForSequenceClassification,
    glue_convert_examples_to_features,
    glue_processors
)

data, info = tensorflow_datasets.load(name='glue/sst2',
                                      data_dir='/tmp/',
                                      with_info=True)

pretrained_weights = 'bert-base-multilingual-uncased'

# Load tokenizer
tokenizer = BertTokenizer.from_pretrained(pretrained_weights)

# recap of input dataset
print(data['train'])
print(tf.data.experimental.cardinality(data['train']))

# Prepare data for BERT
train_dataset = glue_convert_examples_to_features(data['train'], 
                                                  tokenizer, 
                                                  max_length=128, 
                                                  task='sst-2')

# recap of pre processing dataset
print(train_dataset)
print(tf.data.experimental.cardinality(train_dataset))

```
## Expected behavior
I am expecting tf.data.experimental.cardinality to still be able to report the total number of entries after transforming the data with `glue_convert_examples_to_features`

## Environment info
<!-- You can run the command `python transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 2.8.0
- Platform: MacOS 0.14.6 
- Python version: 3.7.5
- Tensorflow version (CPU): 2.2.0rc2 (v2.2.0-rc1-34-ge6e5d6df2a 2.2.0-rc2)


Comments:
This issue might be of interest to @jplu 

Hey @tarrade 

Usually it is not advised to use the cardinality function for several reasons, the biggest two are: 1) it is still experimental, 2) cardinality works only with TF datasets created with `from_tensors` or `from_tensor_slices` which is not the case in the `glue_convert_examples_to_features` function.

If you need to know the size of your dataset from a TF dataset, there are two simple solutions:
```
# "dataset" is the variable that represents your tf.data.dataset
# works only from TF 2.1 because of the as_numpy_iterator() method
len(list(dataset.as_numpy_iterator())
```
Or
```
# works for all the TF versions
dataset.reduce(0, lambda x, _: x + 1)
```

Hi @jplu,

yes, 

this is experimental but so much better that looping over the full dataset just to get the size while such info is almost here for free.

and it doesn't work out of the box with tf.data.Dataset.from_generator

But, tit is already use in the transformers' code: 
https://github.com/huggingface/transformers/blob/master/src/transformers/data/processors/glue.py (line 87)
`
len_examples = tf.data.experimental.cardinality(examples)`

something like that in the right place is the code should work:
`tf.data.experimental.assert_cardinality(len_examples)`



It works in glue.py because `examples` is the direct output of the `tfds.load()` function which returns compatible TF Datasets for `cardinality`.

Getting the size of a TF Dataset if very complicated because of the way it is structured. It is a known problem, many issues are opened in the TF repo because of that. And that's also why the `cardinality| method is still in the experimental module since a very long time now, even much before TF 2.0 was released.

If you are not ok to use one of the two solution I proposed, explain me your use case? Are you using Glue from the `tensorflow_datasets` package? If yes, you have several facilities proposed by this package (https://www.tensorflow.org/datasets/overview)

Hi @jplu,

I understand that using experimental feature of Tensorflow may introduce some instability. This is a faire point and  `tf.data.experimental.assert_cardinality` is only available with TF 2.2.0.

My main points and usecases are:
1- Computing the number of element in a sample is very time consuming since you need to loop over all elements.
2- Normally even if data are coming from `tfds.load()` you need some cleaning, preprocessing steps or maybe you want to resample you train/test/valid sample. In such case the total number from the metadata (info)  will not help since it was changed. This is a normal process is any ML project.
3- In the version of the code was looking at, the length was computed anyway (this doesn't seems to be the case any more with the big clean up from 3 days ago). This was my main argumentation: you compute for any case the total number of even so why not simply assert the cardinality so any sample produced by `glue_convert_examples_to_features` will have the total number of event it contain and for free (no computation required).
4- Now `tf.data.experimental.assert_cardinality(len_examples)` is experimental, require TF 2.2.0 and it the head of the code, the length doesn't seems to be computed any more.
5- One issue is that I soon as I will store the data preprocessed with `glue_convert_examples_to_features` as TFRecord files, then the cardinality will be lost.

Conclusion: I will take care of doing the assert of the cardinality in my code and I hope that when TF 2.2.0 will be the main release and cardinality more stable we could rediscuss this topic.

Good points :)

1. I fully agree with you, it is really painful.
2. If you need to change the content of each dataset (doesn't matter is preprocessing or not)  such as when doing cross-validation, indeed you have to recompute the size.
3. There is currently a project to fully review and rework the data processing part of the lib, so it should be way more convenient to use once done. Until there, indeed, it is a bit of a mess.
4. I was not aware of this new `tf.data.experimental.assert_cardinality(len_examples)` as I did not fully dive in TF 2.2 yet, but looks very interesting, thanks for the hint :)
5. Indeed, the size cannot be computed from TFRecords, which is a real problem IMHO. I hope in future releases it will be much easier to get the size of a dataset ^^

I will be happy to rediscuss about that, very sorry to do no have, sorry that I could not find a suitable solution to your issue.