DataCollatorForLanguageModeling with whole_word_mask=True crashes without offset_mapping
Author: a1nkit
Pull request: no
Labels: name: bug

name: Code agent slop

### System Info

#### **google-colab**
- Transformers: 4.57.3
- Torch: 2.9.0+cpu
- Python: 3.12.12 (main, Oct 10 2025, 08:52:57) [GCC 11.4.0]
- Platform: Linux-6.6.105+-x86_64-with-glibc2.35

---

#### output of `transformers env` command
```
2026-01-20 06:28:11.477925: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2026-01-20 06:28:11.485415: I external/local_xla/xla/tsl/cuda/cudart_stub.cc:32] Could not find cuda drivers on your machine, GPU will not be used.
2026-01-20 06:28:11.530296: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:467] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
E0000 00:00:1768890491.585851     399 cuda_dnn.cc:8579] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
E0000 00:00:1768890491.597823     399 cuda_blas.cc:1407] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
W0000 00:00:1768890491.634226     399 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1768890491.634279     399 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1768890491.634288     399 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
W0000 00:00:1768890491.634296     399 computation_placer.cc:177] computation placer already registered. Please check linkage and avoid linking the same target more than once.
2026-01-20 06:28:11.642835: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:torchao.kernel.intmm:Warning: Detected no triton, on systems without Triton certain kernels will not work
WARNING:tensorflow:From /usr/local/lib/python3.12/dist-packages/transformers/commands/env.py:113: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
WARNING:tensorflow:From /usr/local/lib/python3.12/dist-packages/transformers/commands/env.py:113: is_gpu_available (from tensorflow.python.framework.test_util) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.config.list_physical_devices('GPU')` instead.
2026-01-20 06:28:33.312031: E external/local_xla/xla/stream_executor/cuda/cuda_platform.cc:51] failed call to cuInit: INTERNAL: CUDA error: Failed call to cuInit: UNKNOWN ERROR (303)

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.57.3
- Platform: Linux-6.6.105+-x86_64-with-glibc2.35
- Python version: 3.12.12
- Huggingface_hub version: 0.36.0
- Safetensors version: 0.7.0
- Accelerate version: 1.12.0
- Accelerate config: 	not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.9.0+cpu (NA)
- Tensorflow version (GPU?): 2.19.0 (False)
- Flax version (CPU?/GPU?/TPU?): 0.10.7 (cpu)
- Jax version: 0.7.2
- JaxLib version: 0.7.2
- Using distributed or parallel set-up in script?: <fill in>
```

### Who can help?

@ArthurZucker, @itazap, @SunMarc


### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

## 🐛 Bug Summary
`DataCollatorForLanguageModeling` with `whole_word_mask=True`
crashes with `IndexError` when `offset_mapping` is missing,
instead of raising a clear validation error.

## 🔍 Minimal Reproduction (Colab)
```python
import torch
from transformers import AutoTokenizer, DataCollatorForLanguageModeling

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

data_collator = DataCollatorForLanguageModeling(
    tokenizer=tokenizer,
    mlm=True,
    mlm_probability=1.0,
    whole_word_mask=True,
    mask_replace_prob=1.0,
    random_replace_prob=0.0,
)

text = "The scientist studied unhappiness and cryptography."
inputs = tokenizer(text, return_tensors="pt")

features = [{"input_ids": inputs["input_ids"][0]}]
data_collator(features)
```
### ❌ Actual Behavior

```
IndexError: too many indices for array: array is 0-dimensional
```

### ✅ Expected Behavior

A clear error message indicating that `offset_mapping` (and `special_tokens_mask`) are required when `whole_word_mask=True`, or guidance to use
`DataCollatorForWholeWordMask`.

### 📌 Environment
* Transformers: 4.57.3
* Torch: 2.9.0+cpu
* Python: 3.12.12 (main, Oct 10 2025, 08:52:57) [GCC 11.4.0]
* Platform: Linux-6.6.105+-x86_64-with-glibc2.35     OR       Google Colab (Linux)

### 🔗 Related

* PR: [https://github.com/huggingface/transformers/pull/42144](https://github.com/huggingface/transformers/pull/42144)

### 📌 Recommendation / Design Note

When using Whole Word Masking with BERT:

* `DataCollatorForLanguageModeling(whole_word_mask=True)` **requires** `offset_mapping` and `special_tokens_mask`
* Without offsets, it may crash or behave inconsistently

For standard BERT-style WWM:

* **`DataCollatorForWholeWordMask` is the intended and safer API**
* It does not require offsets
* It is faster, more memory-efficient, and more stable across releases

Using `DataCollatorForLanguageModeling` with WWM:

* Adds overhead (offsets, extra checks)
* Is more fragile to internal changes
* Provides no practical benefit for BERT

➡️ **Recommendation:**
* AI suggests, use `DataCollatorForWholeWordMask` for BERT WWM. (But it is going to be depricated)
* Only use `DataCollatorForLanguageModeling + offsets` if offset-based word alignment is explicitly required.
* `DataCollatorForLanguageModeling(whole_word_mask=True)` **should not require** `offset_mapping` and `special_tokens_mask` as it will require more memory and will make code slower.

### Expected behavior

* `DataCollatorForLanguageModeling(whole_word_mask=True)` **should not require** `offset_mapping` and `special_tokens_mask` as it will require more memory and will make code slower.

* (If above cannot be done) A clear error message indicating that `offset_mapping` (and `special_tokens_mask`) are required when `whole_word_mask=True`, or guidance to use `DataCollatorForWholeWordMask`.

Comments:
Hi! 

I’d like to work on a fix for this issue. My plan is to add an early validation in `DataCollatorForLanguageModeling` so that when `whole_word_mask=True` is used without `offset_mapping`, a clear and informative `ValueError` is raised instead of the current `IndexError`.

Please let me know if this approach sounds reasonable, or if there’s anything you’d recommend adjusting. Thank you!


> Hi!
> 
> I’d like to work on a fix for this issue. My plan is to add an early validation in `DataCollatorForLanguageModeling` so that when `whole_word_mask=True` is used without `offset_mapping`, a clear and informative `ValueError` is raised instead of the current `IndexError`.
> 
> Please let me know if this approach sounds reasonable, or if there’s anything you’d recommend adjusting. Thank you!

@Tanmaygangurde20 According to me, it would be better if you can first think about following: 

`DataCollatorForLanguageModeling(whole_word_mask=True)` should not require `offset_mapping` and `special_tokens_mask` as it will require more memory and will make code slower.




@a1nkit
thank you for the feedback 

Based on your suggestion, I think instead of only adding validation, a better fix would be to remove the hard dependency on `offset_mapping` altogether. My idea is to infer whole-word boundaries directly from the token strings themselves.

### Core Idea

**File:** `src/transformers/data/data_collator.py`  
**Methods:** `torch_mask_tokens()` and `numpy_mask_tokens()`

Rather than relying on `offset_mapping`, I’m thinking we can detect word boundaries from token strings.

For example, with BERT-style tokenization:
````markdown

```python
tokens = ["the", "scientist", "un", "##happiness", "and"]
# word_ids: [0,      1,         2,       2,         3]
#                                        ^ continues "un"
````

Tokens starting with `##` clearly indicate a continuation of the previous word. Using this, we can group subword tokens correctly for whole-word masking without needing character offsets.

### Plan

* Add a helper like `_get_word_ids_from_tokens()` that:

  * Converts `input_ids` back to token strings
  * Detects word boundaries using common subword prefixes (`##`, `Ġ`, `▁`)
* Add a small helper such as `_is_subword_continuation()` to centralize the prefix logic
* Update `torch_mask_tokens()` and `numpy_mask_tokens()` to use this token-based approach instead of `_calc_word_ids_and_prob_mask()`



Does this align with what you had in mind? If this direction looks good, I’m happy to start working on the PR.