Improve robustness of batch_decode in PreTrainedTokenizerBase
Author: AvinashDwivedi
Pull request: no

The current implementation of `batch_decode()` in  
`main/src/transformers/tokenization_utils_base.py`  
does not robustly distinguish between a **single sequence (`list[int]`)** and a **batch of sequences (`list[list[int]]`)**.

As a result, passing a single flat list (or 1D numpy/torch tensor) can cause unexpected behavior — decoding each token ID individually rather than treating the entire list as one sequence.

This proposal introduces a small, **backward-compatible enhancement** that improves type detection and ensures consistent decoding behavior across all supported input types.

---

## Current Implementation (simplified)

```python
def batch_decode(
    self,
    sequences: Union[list[int], list[list[int]], np.ndarray, "torch.Tensor"],
    skip_special_tokens: bool = False,
    clean_up_tokenization_spaces: Optional[bool] = None,
    **kwargs,
) -> list[str]:
    return [
        self.decode(
            seq,
            skip_special_tokens=skip_special_tokens,
            clean_up_tokenization_spaces=clean_up_tokenization_spaces,
            **kwargs,
        )
        for seq in sequences
    ]
```

---

## Problem

If a user calls:

```python
tokenizer.batch_decode([101, 7592, 102])
```

This will iterate over `[101, 7592, 102]` and attempt to decode each integer separately, producing wrong results or raising errors.

Similarly, 1D `np.ndarray` or `torch.Tensor` inputs will behave inconsistently.

---

## Expected Behavior

- A **flat list / 1D tensor / 1D array** should be treated as **one sequence**, returning a single-element list (e.g. `["Hello world"]`).
- A **list of lists / 2D tensor / 2D array** should return a **list of decoded strings** (one per sequence).
- **Empty inputs** should return an empty list.
- All behavior remains **backward compatible** for well-formed batched inputs.

---

## Proposed Fix (safe, minimal change)

A more robust implementation that:

- Detects **1D vs 2D** structure for numpy and torch inputs.  
- Checks **element types** for Python lists/tuples.  
- Keeps existing **kwargs and return structure**.  
- Raises a clear **TypeError** for unsupported types.

**Where:**  
`main/src/transformers/tokenization_utils_base.py`

**What:**  
Modify `batch_decode` inside `PreTrainedTokenizerBase` to:
- Detect single-sequence inputs (1D list/tensor/array) and treat them as batch of size 1.
- Retain the current iteration logic for batched inputs.
- Add validation for input structure and clear error handling.

---

## Proposed Tests

Add to `tests/tokenization/test_tokenization_common.py`:

1. **`test_batch_decode_single_list()`**  
   ✅ `batch_decode([101, 7592, 102])` returns `["hello world"]`

2. **`test_batch_decode_nested_list()`**  
   ✅ `batch_decode([[101, 7592], [102]])` returns `["hello", "world"]`

3. **`test_batch_decode_numpy_tensors()`**  
   ✅ Handles both 1D and 2D numpy/torch tensor inputs consistently.

4. **`test_batch_decode_empty()`**  
   ✅ Returns `[]`

5. **`test_batch_decode_invalid_type()`**  
   ✅ Raises clear `TypeError`.

---

## Benefits

- Prevents silent misbehavior for single-sequence inputs.  
- Ensures consistent behavior across **list**, **numpy**, and **torch** types.  
- Fully **backward compatible** with existing batched usage.  
- Small, **well-isolated** change with clear test coverage.

---

## Suggested Label

`enhancement` / `tokenizers`

---

## Additional Context

This aligns with the **Transformers v5 cleanup and API simplification efforts**,  
where tokenizer APIs (`encode` / `decode`) are being made more explicit and robust.  

This small fix complements that direction by improving one of the most-used utility functions in `PreTrainedTokenizerBase`.

---

## ✅ Summary

**Enhancement proposal:**  
Make `batch_decode` in `main/src/transformers/tokenization_utils_base.py` handle **1D list/tensor inputs safely and consistently**.  

The change is **backward compatible**, adds **clarity**, and supports **better input type handling** for both Python lists and numpy/torch arrays.

Comments:
We have a `decode()` method for decoding a single sequence. If you call `batch_decode()`, we assume you have a batch. I'm not sure this is a bug!