Tokenizer padding does not work when return_tensor="pt"
Author: simeneide
Pull request: no

### System Info

- `transformers` version: 4.36.1
- Platform: Linux-5.15.0-71-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.1.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction
```
from datasets import load_dataset
import transformers
tokenizer = transformers.AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
raw_datasets = load_dataset("glue", "mrpc")
def tokenize_function(example):
    return tokenizer(example["sentence1"], example["sentence2"], truncation=True, return_tensors="pt")
tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)
```
Resulting traceback
```
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File [~/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:748](https://vscode-remote+ssh-002dremote-002bdatacrunch-002dplayground.vscode-resource.vscode-cdn.net/home/simen.eide%40schibsted.com/~/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:748), in BatchEncoding.convert_to_tensors(self, tensor_type, prepend_batch_axis)
    [747](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=746) if not is_tensor(value):
--> [748](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=747)     tensor = as_tensor(value)
    [750](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=749)     # Removing this for now in favor of controlling the shape with `prepend_batch_axis`
    [751](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=750)     # # at-least2d
    [752](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=751)     # if tensor.ndim > 2:
    [753](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=752)     #     tensor = tensor.squeeze(0)
    [754](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=753)     # elif tensor.ndim < 2:
    [755](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=754)     #     tensor = tensor[None, :]

File [~/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:720](https://vscode-remote+ssh-002dremote-002bdatacrunch-002dplayground.vscode-resource.vscode-cdn.net/home/simen.eide%40schibsted.com/~/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:720), in BatchEncoding.convert_to_tensors..as_tensor(value, dtype)
    [719](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=718)     return torch.tensor(np.array(value))
--> [720](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=719) return torch.tensor(value)

ValueError: expected sequence of length 52 at dim 1 (got 77)

The above exception was the direct cause of the following exception:

ValueError                                Traceback (most recent call last)
[/home/simen.eide](https://vscode-remote+ssh-002dremote-002bdatacrunch-002dplayground.vscode-resource.vscode-cdn.net/home/simen.eide)@schibsted.com/.local/lib/python3.10/site-packages/transformers/data/Untitled-1.py in line 8
      [6](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/data/Untitled-1.py?line=5) def tokenize_function(example):
      [7](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/data/Untitled-1.py?line=6)     return tokenizer(example["sentence1"], example["sentence2"], truncation=True, return_tensors="pt")
----> [8](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/data/Untitled-1.py?line=7) tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)

File [~/.local/lib/python3.10/site-packages/datasets/dataset_dict.py:855](https://vscode-remote+ssh-002dremote-002bdatacrunch-002dplayground.vscode-resource.vscode-cdn.net/home/simen.eide%40schibsted.com/~/.local/lib/python3.10/site-packages/datasets/dataset_dict.py:855), in DatasetDict.map(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_names, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, desc)
    [852](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=851) if cache_file_names is None:
    [853](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=852)     cache_file_names = {k: None for k in self}
    [854](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=853) return DatasetDict(
--> [855](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=854)     {
    [856](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=855)         k: dataset.map(
    [857](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=856)             function=function,
    [858](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=857)             with_indices=with_indices,
    [859](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=858)             with_rank=with_rank,
    [860](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=859)             input_columns=input_columns,
    [861](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=860)             batched=batched,
    [862](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=861)             batch_size=batch_size,
    [863](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=862)             drop_last_batch=drop_last_batch,
    [864](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=863)             remove_columns=remove_columns,
    [865](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=864)             keep_in_memory=keep_in_memory,
    [866](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=865)             load_from_cache_file=load_from_cache_file,
    [867](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=866)             cache_file_name=cache_file_names[k],
    [868](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=867)             writer_batch_size=writer_batch_size,
    [869](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=868)             features=features,
    [870](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=869)             disable_nullable=disable_nullable,
    [871](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=870)             fn_kwargs=fn_kwargs,
    [872](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=871)             num_proc=num_proc,
    [873](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=872)             desc=desc,
    [874](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=873)         )
    [875](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=874)         for k, dataset in self.items()
    [876](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=875)     }
    [877](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=876) )

File [~/.local/lib/python3.10/site-packages/datasets/dataset_dict.py:856](https://vscode-remote+ssh-002dremote-002bdatacrunch-002dplayground.vscode-resource.vscode-cdn.net/home/simen.eide%40schibsted.com/~/.local/lib/python3.10/site-packages/datasets/dataset_dict.py:856), in (.0)
    [852](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=851) if cache_file_names is None:
    [853](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=852)     cache_file_names = {k: None for k in self}
    [854](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=853) return DatasetDict(
    [855](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=854)     {
--> [856](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=855)         k: dataset.map(
    [857](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=856)             function=function,
    [858](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=857)             with_indices=with_indices,
    [859](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=858)             with_rank=with_rank,
    [860](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=859)             input_columns=input_columns,
    [861](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=860)             batched=batched,
    [862](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=861)             batch_size=batch_size,
    [863](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=862)             drop_last_batch=drop_last_batch,
    [864](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=863)             remove_columns=remove_columns,
    [865](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=864)             keep_in_memory=keep_in_memory,
    [866](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=865)             load_from_cache_file=load_from_cache_file,
    [867](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=866)             cache_file_name=cache_file_names[k],
    [868](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=867)             writer_batch_size=writer_batch_size,
    [869](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=868)             features=features,
    [870](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=869)             disable_nullable=disable_nullable,
    [871](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=870)             fn_kwargs=fn_kwargs,
    [872](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=871)             num_proc=num_proc,
    [873](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=872)             desc=desc,
    [874](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=873)         )
    [875](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=874)         for k, dataset in self.items()
    [876](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=875)     }
    [877](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/dataset_dict.py?line=876) )

File [~/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py:591](https://vscode-remote+ssh-002dremote-002bdatacrunch-002dplayground.vscode-resource.vscode-cdn.net/home/simen.eide%40schibsted.com/~/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py:591), in transmit_tasks..wrapper(*args, **kwargs)
    [589](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=588)     self: "Dataset" = kwargs.pop("self")
    [590](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=589) # apply actual function
--> [591](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=590) out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    [592](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=591) datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    [593](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=592) for dataset in datasets:
    [594](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=593)     # Remove task templates if a column mapping of the template is no longer valid

File [~/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py:556](https://vscode-remote+ssh-002dremote-002bdatacrunch-002dplayground.vscode-resource.vscode-cdn.net/home/simen.eide%40schibsted.com/~/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py:556), in transmit_format..wrapper(*args, **kwargs)
    [549](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=548) self_format = {
    [550](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=549)     "type": self._format_type,
    [551](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=550)     "format_kwargs": self._format_kwargs,
    [552](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=551)     "columns": self._format_columns,
    [553](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=552)     "output_all_columns": self._output_all_columns,
    [554](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=553) }
    [555](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=554) # apply actual function
--> [556](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=555) out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
    [557](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=556) datasets: List["Dataset"] = list(out.values()) if isinstance(out, dict) else [out]
    [558](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=557) # re-apply format to the output

File [~/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py:3089](https://vscode-remote+ssh-002dremote-002bdatacrunch-002dplayground.vscode-resource.vscode-cdn.net/home/simen.eide%40schibsted.com/~/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py:3089), in Dataset.map(self, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, load_from_cache_file, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, num_proc, suffix_template, new_fingerprint, desc)
   [3082](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3081) if transformed_dataset is None:
   [3083](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3082)     with logging.tqdm(
   [3084](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3083)         disable=not logging.is_progress_bar_enabled(),
   [3085](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3084)         unit=" examples",
   [3086](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3085)         total=pbar_total,
   [3087](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3086)         desc=desc or "Map",
   [3088](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3087)     ) as pbar:
-> [3089](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3088)         for rank, done, content in Dataset._map_single(**dataset_kwargs):
   [3090](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3089)             if done:
   [3091](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3090)                 shards_done += 1

File [~/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py:3466](https://vscode-remote+ssh-002dremote-002bdatacrunch-002dplayground.vscode-resource.vscode-cdn.net/home/simen.eide%40schibsted.com/~/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py:3466), in Dataset._map_single(shard, function, with_indices, with_rank, input_columns, batched, batch_size, drop_last_batch, remove_columns, keep_in_memory, cache_file_name, writer_batch_size, features, disable_nullable, fn_kwargs, new_fingerprint, rank, offset)
   [3462](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3461) indices = list(
   [3463](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3462)     range(*(slice(i, i + batch_size).indices(shard.num_rows)))
   [3464](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3463) )  # Something simpler?
   [3465](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3464) try:
-> [3466](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3465)     batch = apply_function_on_filtered_inputs(
   [3467](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3466)         batch,
   [3468](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3467)         indices,
   [3469](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3468)         check_same_num_examples=len(shard.list_indexes()) > 0,
   [3470](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3469)         offset=offset,
   [3471](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3470)     )
   [3472](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3471) except NumExamplesMismatchError:
   [3473](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3472)     raise DatasetTransformationNotAllowedError(
   [3474](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3473)         "Using `.map` in batched mode on a dataset with attached indexes is allowed only if it doesn't create or remove existing examples. You can first run `.drop_index() to remove your index and then re-add it."
   [3475](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3474)     ) from None

File [~/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py:3345](https://vscode-remote+ssh-002dremote-002bdatacrunch-002dplayground.vscode-resource.vscode-cdn.net/home/simen.eide%40schibsted.com/~/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py:3345), in Dataset._map_single..apply_function_on_filtered_inputs(pa_inputs, indices, check_same_num_examples, offset)
   [3343](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3342) if with_rank:
   [3344](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3343)     additional_args += (rank,)
-> [3345](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3344) processed_inputs = function(*fn_args, *additional_args, **fn_kwargs)
   [3346](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3345) if isinstance(processed_inputs, LazyDict):
   [3347](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3346)     processed_inputs = {
   [3348](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3347)         k: v for k, v in processed_inputs.data.items() if k not in processed_inputs.keys_to_format
   [3349](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/datasets/arrow_dataset.py?line=3348)     }

[/home/simen.eide](https://vscode-remote+ssh-002dremote-002bdatacrunch-002dplayground.vscode-resource.vscode-cdn.net/home/simen.eide)@schibsted.com/.local/lib/python3.10/site-packages/transformers/data/Untitled-1.py in line 7, in tokenize_function(example)
      [6](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/data/Untitled-1.py?line=5) def tokenize_function(example):
----> [7](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/data/Untitled-1.py?line=6)     return tokenizer(example["sentence1"], example["sentence2"], truncation=True, return_tensors="pt")

File [~/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:2802](https://vscode-remote+ssh-002dremote-002bdatacrunch-002dplayground.vscode-resource.vscode-cdn.net/home/simen.eide%40schibsted.com/~/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:2802), in PreTrainedTokenizerBase.__call__(self, text, text_pair, text_target, text_pair_target, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
   [2800](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2799)     if not self._in_target_context_manager:
   [2801](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2800)         self._switch_to_input_mode()
-> [2802](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2801)     encodings = self._call_one(text=text, text_pair=text_pair, **all_kwargs)
   [2803](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2802) if text_target is not None:
   [2804](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2803)     self._switch_to_target_mode()

File [~/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:2888](https://vscode-remote+ssh-002dremote-002bdatacrunch-002dplayground.vscode-resource.vscode-cdn.net/home/simen.eide%40schibsted.com/~/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:2888), in PreTrainedTokenizerBase._call_one(self, text, text_pair, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
   [2883](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2882)         raise ValueError(
   [2884](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2883)             f"batch length of `text`: {len(text)} does not match batch length of `text_pair`:"
   [2885](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2884)             f" {len(text_pair)}."
   [2886](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2885)         )
   [2887](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2886)     batch_text_or_text_pairs = list(zip(text, text_pair)) if text_pair is not None else text
-> [2888](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2887)     return self.batch_encode_plus(
   [2889](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2888)         batch_text_or_text_pairs=batch_text_or_text_pairs,
   [2890](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2889)         add_special_tokens=add_special_tokens,
   [2891](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2890)         padding=padding,
   [2892](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2891)         truncation=truncation,
   [2893](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2892)         max_length=max_length,
   [2894](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2893)         stride=stride,
   [2895](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2894)         is_split_into_words=is_split_into_words,
   [2896](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2895)         pad_to_multiple_of=pad_to_multiple_of,
   [2897](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2896)         return_tensors=return_tensors,
   [2898](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2897)         return_token_type_ids=return_token_type_ids,
   [2899](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2898)         return_attention_mask=return_attention_mask,
   [2900](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2899)         return_overflowing_tokens=return_overflowing_tokens,
   [2901](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2900)         return_special_tokens_mask=return_special_tokens_mask,
   [2902](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2901)         return_offsets_mapping=return_offsets_mapping,
   [2903](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2902)         return_length=return_length,
   [2904](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2903)         verbose=verbose,
   [2905](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2904)         **kwargs,
   [2906](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2905)     )
   [2907](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2906) else:
   [2908](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2907)     return self.encode_plus(
   [2909](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2908)         text=text,
   [2910](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2909)         text_pair=text_pair,
   (...)
   [2926](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2925)         **kwargs,
   [2927](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=2926)     )

File [~/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:3079](https://vscode-remote+ssh-002dremote-002bdatacrunch-002dplayground.vscode-resource.vscode-cdn.net/home/simen.eide%40schibsted.com/~/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:3079), in PreTrainedTokenizerBase.batch_encode_plus(self, batch_text_or_text_pairs, add_special_tokens, padding, truncation, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose, **kwargs)
   [3069](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3068) # Backward compatibility for 'truncation_strategy', 'pad_to_max_length'
   [3070](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3069) padding_strategy, truncation_strategy, max_length, kwargs = self._get_padding_truncation_strategies(
   [3071](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3070)     padding=padding,
   [3072](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3071)     truncation=truncation,
   (...)
   [3076](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3075)     **kwargs,
   [3077](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3076) )
-> [3079](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3078) return self._batch_encode_plus(
   [3080](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3079)     batch_text_or_text_pairs=batch_text_or_text_pairs,
   [3081](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3080)     add_special_tokens=add_special_tokens,
   [3082](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3081)     padding_strategy=padding_strategy,
   [3083](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3082)     truncation_strategy=truncation_strategy,
   [3084](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3083)     max_length=max_length,
   [3085](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3084)     stride=stride,
   [3086](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3085)     is_split_into_words=is_split_into_words,
   [3087](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3086)     pad_to_multiple_of=pad_to_multiple_of,
   [3088](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3087)     return_tensors=return_tensors,
   [3089](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3088)     return_token_type_ids=return_token_type_ids,
   [3090](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3089)     return_attention_mask=return_attention_mask,
   [3091](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3090)     return_overflowing_tokens=return_overflowing_tokens,
   [3092](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3091)     return_special_tokens_mask=return_special_tokens_mask,
   [3093](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3092)     return_offsets_mapping=return_offsets_mapping,
   [3094](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3093)     return_length=return_length,
   [3095](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3094)     verbose=verbose,
   [3096](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3095)     **kwargs,
   [3097](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=3096) )

File [~/.local/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py:552](https://vscode-remote+ssh-002dremote-002bdatacrunch-002dplayground.vscode-resource.vscode-cdn.net/home/simen.eide%40schibsted.com/~/.local/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py:552), in PreTrainedTokenizerFast._batch_encode_plus(self, batch_text_or_text_pairs, add_special_tokens, padding_strategy, truncation_strategy, max_length, stride, is_split_into_words, pad_to_multiple_of, return_tensors, return_token_type_ids, return_attention_mask, return_overflowing_tokens, return_special_tokens_mask, return_offsets_mapping, return_length, verbose)
    [550](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py?line=549) for input_ids in sanitized_tokens["input_ids"]:
    [551](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py?line=550)     self._eventual_warn_about_too_long_sequence(input_ids, max_length, verbose)
--> [552](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_fast.py?line=551) return BatchEncoding(sanitized_tokens, sanitized_encodings, tensor_type=return_tensors)

File [~/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:223](https://vscode-remote+ssh-002dremote-002bdatacrunch-002dplayground.vscode-resource.vscode-cdn.net/home/simen.eide%40schibsted.com/~/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:223), in BatchEncoding.__init__(self, data, encoding, tensor_type, prepend_batch_axis, n_sequences)
    [219](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=218)     n_sequences = encoding[0].n_sequences
    [221](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=220) self._n_sequences = n_sequences
--> [223](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=222) self.convert_to_tensors(tensor_type=tensor_type, prepend_batch_axis=prepend_batch_axis)

File [~/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:764](https://vscode-remote+ssh-002dremote-002bdatacrunch-002dplayground.vscode-resource.vscode-cdn.net/home/simen.eide%40schibsted.com/~/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py:764), in BatchEncoding.convert_to_tensors(self, tensor_type, prepend_batch_axis)
    [759](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=758)         if key == "overflowing_tokens":
    [760](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=759)             raise ValueError(
    [761](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=760)                 "Unable to create tensor returning overflowing tokens of different lengths. "
    [762](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=761)                 "Please see if a fast version of this tokenizer is available to have this feature available."
    [763](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=762)             ) from e
--> [764](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=763)         raise ValueError(
    [765](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=764)             "Unable to create tensor, you should probably activate truncation and/or padding with"
    [766](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=765)             " 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your"
    [767](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=766)             f" features (`{key}` in this case) have excessive nesting (inputs type `list` where type `int` is"
    [768](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=767)             " expected)."
    [769](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=768)         ) from e
    [771](file:///home/simen.eide%40schibsted.com/.local/lib/python3.10/site-packages/transformers/tokenization_utils_base.py?line=770) return self

ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (`input_ids` in this case) have excessive nesting (inputs type `list` where type `int` is expected).
```

### Expected behavior

I am following the manual on dynamic padding here: https://huggingface.co/learn/nlp-course/chapter3/2?fw=pt#dynamic-padding

When it returns lists it all is fine, but the padding fails when I ask the tokenizer to return "pt".

Comments:
Could you share a traceback of the error? 


Sorry, it is updated now :)

Also related but not sure if it is the same. When using DataCollator on batches with keys that are not inputIds it fails:
https://huggingface.co/docs/transformers/main_classes/data_collator#transformers.DataCollatorWithPadding

it calls tokenizer.pad inside which does seem to break e.g.

```
tokenizer = transformers.AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
tokenizer.pad_token = tokenizer.eos_token
tokenizer.pad([{'input_ids' : [1,2], 'label' : [1,2]}, {'input_ids' : [1,2,3], 'label' : [1,2,3]},], return_tensors="pt" )
# ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (`label` in this case) have excessive nesting (inputs type `list` where type `int` is expected).
```

Few things here. 
- you need to pad the data. For that you need to add a padding token 
- you need to set `padding=True` the following worked for me:
```python
 def tokenize_function(example):
    return tokenizer(example["sentence1"], example["sentence2"], padding = True, truncation=True, return_tensors="pt")
tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)
```
- why you [need to pad ](https://huggingface.co/docs/transformers/pad_truncation)as well? truncation only truncates the longest sequence, the others have to be padded. 

[This](https://discuss.huggingface.co/t/the-datasets-map-method-doesnt-keep-tensor-format-from-tokenizer/25482) as well for the output format. 

Hey, thanks for looking into this. 
I am (as stated) following the dynamic padding tutorial (https://huggingface.co/learn/nlp-course/chapter3/2?fw=pt#dynamic-padding) and the code is taken from there, just that I added `return_tensors="pt"`.

The reason I do not want to pad here is because I want to pad in a collate_fn function later (to get dynamic padding). But I would prefer to use tensors instead of lists when tokenizing. The error comes from the tokenizer.pad (where i added an example in the comments).

Is this function supposed to work when the tokenizer returns lists but not with tensors?

Of course, you cannot create a tensor with samples of different sizes. That is why we have to pad in that context

I dont think you understood what I meant (i want to use dynamic padding in a collate_fn function), but Ive found another solution :) Thanks

Alright ! Feel free to share it here for the community 🤗 

Yes, I basically had to write my own DataCollator. The problem is that `tokenizer.pad` will not work in the data collator if there are features other than input_ids & attention_mask in the dataset.

This issue summarizes it fairly well:
https://github.com/huggingface/transformers/issues/20182

My solution is the following modification to the DataCollatorWithPadding:

```
from dataclasses import dataclass
from random import randint
from typing import Any, Callable, Dict, List, NewType, Optional, Tuple, Union
from transformers.utils import PaddingStrategy
from transformers import PreTrainedTokenizerBase

@dataclass
class DataCollatorWithPadding:
    """
    Data collator that will dynamically pad the inputs received.

    Args:
        tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]):
            The tokenizer used for encoding the data.
        padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`):
            Select a strategy to pad the returned sequences (according to the model's padding side and padding index)
            among:

            - `True` or `'longest'` (default): Pad to the longest sequence in the batch (or no padding if only a single
              sequence is provided).
            - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum
              acceptable input length for the model if that argument is not provided.
            - `False` or `'do_not_pad'`: No padding (i.e., can output a batch with sequences of different lengths).
        max_length (`int`, *optional*):
            Maximum length of the returned list and optionally padding length (see above).
        pad_to_multiple_of (`int`, *optional*):
            If set will pad the sequence to a multiple of the provided value.

            This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >=
            7.5 (Volta).
        return_tensors (`str`, *optional*, defaults to `"pt"`):
            The type of Tensor to return. Allowable values are "np", "pt" and "tf".
    """

    tokenizer : PreTrainedTokenizerBase
    padding: Union[bool, str, PaddingStrategy] = True
    max_length: Optional[int] = None
    pad_to_multiple_of: Optional[int] = None
    return_tensors: str = "pt"

    def __call__(self, features: List[Dict[str, Any]]) -> Dict[str, Any]:
        padding_features = {key : val for key, val in features.items() if key in ['input_ids','attention_mask']}
        batch = self.tokenizer.pad(
            padding_features,
            padding=self.padding,
            max_length=self.max_length,
            pad_to_multiple_of=self.pad_to_multiple_of,
            return_tensors=self.return_tensors,
        )

        batch['labels'] = self.tokenizer.pad(
            {'input_ids' : features['labels']},
            padding=self.padding,
            max_length=self.max_length,
            pad_to_multiple_of=self.pad_to_multiple_of,
            return_tensors=self.return_tensors,
        )['input_ids']
        
        for key, val in features.items():
            if key not in ['input_ids','attention_mask','labels']:
                batch[key] = val

        return batch
```

@simeneide your method is good. And if DataCollatorWithPadding in transformers only pad input_ids and attention_masks. It's not very useful. @ArthurZucker 

@simeneide I am using transformers 4.62.3. The features is of type List[Dict[str, Any]]. So I modified a little bit.

```
from dataclasses import dataclass
from random import randint
from typing import Any, Callable, Dict, List, NewType, Optional, Tuple, Union
from transformers.utils import PaddingStrategy
from transformers import PreTrainedTokenizerBase, BatchEncoding

@dataclass
class MyDataCollatorWithPadding:
    """
    Data collator that will dynamically pad the inputs received.

    Args:
        tokenizer ([`PreTrainedTokenizer`] or [`PreTrainedTokenizerFast`]):
            The tokenizer used for encoding the data.
        padding (`bool`, `str` or [`~utils.PaddingStrategy`], *optional*, defaults to `True`):
            Select a strategy to pad the returned sequences (according to the model's padding side and padding index)
            among:

            - `True` or `'longest'` (default): Pad to the longest sequence in the batch (or no padding if only a single
              sequence is provided).
            - `'max_length'`: Pad to a maximum length specified with the argument `max_length` or to the maximum
              acceptable input length for the model if that argument is not provided.
            - `False` or `'do_not_pad'`: No padding (i.e., can output a batch with sequences of different lengths).
        max_length (`int`, *optional*):
            Maximum length of the returned list and optionally padding length (see above).
        pad_to_multiple_of (`int`, *optional*):
            If set will pad the sequence to a multiple of the provided value.

            This is especially useful to enable the use of Tensor Cores on NVIDIA hardware with compute capability >=
            7.5 (Volta).
        return_tensors (`str`, *optional*, defaults to `"pt"`):
            The type of Tensor to return. Allowable values are "np", "pt" and "tf".
    """

    tokenizer : PreTrainedTokenizerBase
    padding: Union[bool, str, PaddingStrategy] = True
    max_length: Optional[int] = None
    pad_to_multiple_of: Optional[int] = None
    return_tensors: str = "pt"

    def __call__(self, features: List[Dict[str, Any]]) -> Dict[str, Any]:
        padding_features = [{key : val for key, val in row.items() if key in ['input_ids','attention_mask']} for row in features]
        
        batch = self.tokenizer.pad(
            padding_features,
            padding=self.padding,
            max_length=self.max_length,
            pad_to_multiple_of=self.pad_to_multiple_of,
            return_tensors=None,
        )

        batch['labels'] = self.tokenizer.pad(
            [{'input_ids' : row['labels']} for row in features],
            padding=self.padding,
            max_length=self.max_length,
            pad_to_multiple_of=self.pad_to_multiple_of,
            return_tensors=None,
        )['input_ids']
        
 
        for row in features:
            for key, value in row.items():
                if key in ['input_ids','attention_mask','labels']:
                    continue
                if key not in batch:
                    batch[key] = []
                batch[key].append(value)

        return BatchEncoding(batch, tensor_type=self.return_tensors)

```

@fancyerii sorry for the inconvenience, an error raising proposing a fix on how to pad if there are more than 2 inputs might be nice indeed! 

I ran into the same problem in my scenario of using a custom data set to fine-tune the LLAMA2 model, and I used the following function to process each piece of data:
```
def tokenize_add_label(example):
        prompt = tokenizer.encode(
            tokenizer.bos_token + example["prompt"], add_special_tokens=False
        )
        action = tokenizer.encode(
            example["action"] + tokenizer.eos_token, add_special_tokens=False
        )

        example = {
            "input_ids": prompt + action,
            "attention_mask": [1] * (len(prompt) + len(action)),
            "labels": [-100] * len(prompt) + action,
        }
```
I want to train the model using 'DataCollatorWithPadding' to fill in the data for each batch during training and get the same error:
```
data_collator = DataCollatorWithPadding(tokenizer=tokenizer)
trainer = Trainer(
            tokenizer=tokenizer,
            model=model,
            args=training_args,
            train_dataset=train_dataset,
            eval_dataset=eval_dataset,
            data_collator=data_collator,
            callbacks=[profiler_callback] if enable_profiler else [],
        )
trainer.train()
lib/python3.9 / site - packages/transformers/tokenization_utils_base. Py ", line 764, in convert_to_tensors
raise ValueError(
ValueError:  Unable to create tensor,  you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors  with the same length. Perhaps your features (`labels` in this case) have excessive nesting (inputs type `list` where  type `int` is expected).
```

Same error here:
```
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM2-135M")

batch = {
    "input_ids": [[1, 2, 3], [4, 5, 6, 7]],
    "attention_mask": [[1, 1, 1], [1, 1, 1, 1]],
    "labels": [[1,2,3], [4,5,6,7]],
}
tokenizer.pad_token = tokenizer.eos_token
tokenizer.pad(batch, return_tensors="pt")
```
Removing the `labels` from the dict does work.

cc @Rocketknight1 as you worked on batch encoding recently