transformers/examples/tensorflow/tokenclassification: Error at prepare_tf_dataset() using demo code with default parameters.
Author: adaml-iri
Pull request: no

### System Info

- `transformers` version: 4.26.0.dev0
- Platform: Windows-10-10.0.19044-SP0
- Python version: 3.9.13
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): not installed (NA)
- Tensorflow version (GPU?): 2.11.0 (False)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@Rocketknight1

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. git clone https://github.com/huggingface/transformers
2. cd transformers
3. pip install .
4. cd examples\tensorflow\token-classification
5. pip install -r requirements.txt
6. python run_ner.py \
  --model_name_or_path bert-base-uncased \
  --dataset_name conll2003 \
  --output_dir /tmp/test-ner

### Expected behavior

Expected example to fine-tunes BERT on CoNLL-2003.

Comments:
When I attempt to run demo I get following error:

Traceback (most recent call last):
  File "D:\transformer\foo\lib\site-packages\transformers\tokenization_utils_base.py", line 717, in convert_to_tensors
    tensor = as_tensor(value)
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 1 dimensions. The detected shape was (2,) + inhomogeneous part.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\transformer\transformers\examples\tensorflow\token-classification\run_ner.py", line 592, in <module>
    main()
  File "D:\transformer\transformers\examples\tensorflow\token-classification\run_ner.py", line 415, in main
    tf_train_dataset = model.prepare_tf_dataset(
  File "D:\transformer\foo\lib\site-packages\transformers\modeling_tf_utils.py", line 1384, in prepare_tf_dataset
    tf_dataset = dataset.to_tf_dataset(
  File "D:\transformer\foo\lib\site-packages\datasets\arrow_dataset.py", line 405, in to_tf_dataset
    output_signature, columns_to_np_types = dataset._get_output_signature(
  File "D:\transformer\foo\lib\site-packages\datasets\arrow_dataset.py", line 258, in _get_output_signature
    test_batch = collate_fn(test_batch, **collate_fn_args)
  File "D:\transformer\foo\lib\site-packages\transformers\data\data_collator.py", line 43, in __call__
    return self.tf_call(features)
  File "D:\transformer\foo\lib\site-packages\transformers\data\data_collator.py", line 347, in tf_call
    batch = self.tokenizer.pad(
  File "D:\transformer\foo\lib\site-packages\transformers\tokenization_utils_base.py", line 3017, in pad
    return BatchEncoding(batch_outputs, tensor_type=return_tensors)
  File "D:\transformer\foo\lib\site-packages\transformers\tokenization_utils_base.py", line 210, in __init__
    self.convert_to_tensors(tensor_type=tensor_type, prepend_batch_axis=prepend_batch_axis)
  File "D:\transformer\foo\lib\site-packages\transformers\tokenization_utils_base.py", line 733, in convert_to_tensors
    raise ValueError(
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (`labels` in this case) have excessive nesting (inputs type `list` where type `int` is expected).

My problem seems to be with prepare_tf_dataset() when run locally in python script.
I have noticed that when I try notebooks/examples/token_classification-tf.ipynb in Google Colab everything works fine. 

@adaml-iri 

**How to solve the issue :** 

Add "--pad_to_max_length True" as an argument, so to start training you need to write, 
`python run_ner.py --model_name_or_path bert-base-uncased --dataset_name conll2003 --output_dir /tmp/test-ner --pad_to_max_length True`

**Why is it happening ?**

It's due to shape mismatch in the training sample's labels.(In line 717 in https://github.com/huggingface/transformers/blob/main/src/transformers/tokenization_utils_base.py) where the code is trying to convert the labels to numpy array using `np.asarray` but all the examples doesn't have labels with same shape so it's happening. 


**Here is the output you might see if this is resolved :** 

...
All model checkpoint layers were used when initializing TFBertForTokenClassification.

Some layers of TFBertForTokenClassification were not initialized from the model checkpoint at bert-base-uncased and are newly initialized: ['classifier']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
You're using a BertTokenizerFast tokenizer. Please note that with a fast tokenizer, using the `__call__` method is faster than using a method to encode the text followed by a call to the `pad` method to get a padded encoding.
No loss specified in compile() - the model's internal loss computation will be used as the loss. Don't panic - this is a common way to train TensorFlow models in Transformers! To disable this behaviour please pass a loss argument, or explicitly pass `loss=None` if you do not want your model to compute a loss.
***** Running training *****
  Num examples = 14041
  Num Epochs = 3.0
  Instantaneous batch size per device = 8
  Total train batch size = 8
2023-01-07 00:49:52.596921: W tensorflow/core/framework/dataset.cc:769] Input of GeneratorDatasetOp::Dataset will not be optimized because the dataset does not implement the AsGraphDefInternal() method needed to apply optimizations.
Epoch 1/3
 110/1755 [>.............................] - ETA: 2:28:11 - loss: 0.3548



Let me know if you managed to resolve it or not, 

Thanks, 
susnato.

Thank you for your quick response. Everything works now.
Much appreciated 👍 

Hi @adaml-iri - sorry for the delay with dealing with this! I'm glad your issue got resolved, but when I run the code locally I don't get the same issue, and I don't think that example should require `pad_to_max_length` to be set to work. Can you try updating `datasets` as well with `pip install --upgrade datasets` and then checking if the issue persists?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.