Issue while training on custom data
Hello ! I tried fine tuning the model on my custom data and I encountered this error : ---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py in convert_to_tensors(self, tensor_type, prepend_batch_axis)
747 if not is_tensor(value):
--> 748 tensor = as_tensor(value)
749
14 frames
ValueError: expected sequence of length 15 at dim 1 (got 20)
The above exception was the direct cause of the following exception:
ValueError Traceback (most recent call last)
/usr/local/lib/python3.10/dist-packages/transformers/tokenization_utils_base.py in convert_to_tensors(self, tensor_type, prepend_batch_axis)
762 "Please see if a fast version of this tokenizer is available to have this feature available."
763 ) from e
--> 764 raise ValueError(
765 "Unable to create tensor, you should probably activate truncation and/or padding with"
766 " 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your"
ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (labels in this case) have excessive nesting (inputs type list where type int is expected). anyone knows why? any help is much appreciated :)