Repository: huggingface/transformers

Issue:
DataCollatorForWholeWordMask only works for BERT, and nothing is said in the docstring.
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version:
- Platform:
- Python version:
- PyTorch version (GPU?):
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
@patrickvonplaten @LysandreJik @patil-suraj @sgugger

## Information

Model I am using (Bert, XLNet ...): DBERTA (V1) BASE

The problem arises when using:
* [x] the official example scripts: (give details below): DataCollatorForWholeWordMask
* [ ] my own modified scripts: (give details below)

The tasks I am working on is:
* [ ] an official GLUE/SQUaD task: (give the name)
* [ ] my own task or dataset: (give details below)

## To reproduce

The DataCollatorForWholeWordMask, that should be used for pre-training a Roberta model, or a Deberta model for example (as you don't have a SpanCollator), only works for BERT, and one needs to look the details in the collator code to notice this. I've been training a language model from scratch for weeks now, just to notice yesterday that your collator for WholeWordMask is wrong and only works for BERT. 

Steps to reproduce the behavior:

1. Try to use the DataCollatorForWholeWordMask with any model that is not BERT.

## Expected behavior

A data collator that is included in your data collators should work generally for any model, not only for BERT. Or at least, in the Docstring it should be clear that one will waste huge amounts of money if using this collator for other models that are not BERT. This being said, I would like to know how could I use the word_ids from the tokenizer to do this, as with the TokenClassification example you provide here: https://colab.research.google.com/github/huggingface/notebooks/blob/master/examples/text_classification.ipynb#scrollTo=MOsHUjgdIrIW 
In this example the extension of the token labels doesn't depend on the continuation token having "##" at the beginning, but uses the word ids from the FastTokenizer. I think the DataCollatorForWholeWordMask should work generally, at least for all fast tokenizers, not only for BERT. For my case, I would like to know what can I do to at least train a little bit more with the correct objective, not with normal MLM but with WWMLM. 


Discussion:
Yes merging it was a mistake. It will be removed when we have something better in the future.
@sgugger Could you please tell me how could I adapt it for a general fast tokenizer? Or at least how would you do it for a ByteBPETokenizer like Roberta's or Deberta's?
I haven't dug into this, but it should probably leverage the `word_ids` the fast tokenizer provide to be more general.
This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.
I just ran into the same problem. Is somebody working on this?

I need a language modeling data collator for RoBERTa-style tokenizers and might as well try my hand at providing an extensible, general implementation that issues proper warnings if used on yet-unsupported tokenizer classes, if there's interest.
The problem is that after passing through datasets, the objects are dicts, not BatchEncoding, therefore they don't have the word_ids() method, and without that we cannot generalize Whole Word Masking. One solution is to pre tokenize and pre process the dataset inside the function you put in the datasets map, however you disable dynamic batching which is a key improvement of Roberta with respect to Bert.
Thank you for elaborating!

Similarly to the implementation for BERT tokenizers in the current `DataCollatorForWholeWordMasking`, it is possible to obtain a word start mask for RoBERTa tokenizers by decoding every token in the collator by using something like this:

```python
def _word_starts(self, inputs: torch.Tensor) -> torch.Tensor:
    is_word_start = torch.full_like(inputs, fill_value=False)
    for i, example in enumerate(torch.split(inputs, split_size_or_sections=1, dim=0)):
        line_mask = torch.tensor([self.tokenizer.decode([t]).startswith(" ") for t in example.flatten().tolist()
                                  if t != self.tokenizer.pad_token_id])
        is_word_start[i, 0:line_mask.shape[0]] = line_mask
    return is_word_start
```

I believe that this is accurate if the tokenizer is initialized with `add_prefix_space=True`, otherwise the first word is missing, which is probably acceptable in most circumstances.

If this method is correct, it could be extended to BART tokenizers, where the condition for the first token of a word is `not tokenizer.decode([t]).startswith('##')`. I'm not sure whether this is a path one wants to take here, though.

Resolving patch:
diff --git a/src/transformers/data/data_collator.py b/src/transformers/data/data_collator.py
--- a/src/transformers/data/data_collator.py
+++ b/src/transformers/data/data_collator.py
@@ -22,6 +22,7 @@
 
 from ..file_utils import PaddingStrategy
 from ..modeling_utils import PreTrainedModel
+from ..models.bert import BertTokenizer, BertTokenizerFast
 from ..tokenization_utils_base import BatchEncoding, PreTrainedTokenizerBase
 
 
@@ -395,10 +396,17 @@ def mask_tokens(
 @dataclass
 class DataCollatorForWholeWordMask(DataCollatorForLanguageModeling):
     """
-    Data collator used for language modeling.
+    Data collator used for language modeling that masks entire words.
 
     - collates batches of tensors, honoring their tokenizer's pad_token
     - preprocesses batches for masked language modeling
+
+    .. note::
+
+        This collator relies on details of the implementation of subword tokenization by
+        :class:`~transformers.BertTokenizer`, specifically that subword tokens are prefixed with `##`. For tokenizers
+        that do not adhere to this scheme, this collator will produce an output that is roughly equivalent to
+        :class:`.DataCollatorForLanguageModeling`.
     """
 
     def __call__(
@@ -435,6 +443,11 @@ def _whole_word_mask(self, input_tokens: List[str], max_predictions=512):
         """
         Get 0/1 labels for masked tokens with whole word mask proxy
         """
+        if not isinstance(self.tokenizer, (BertTokenizer, BertTokenizerFast)):
+            warnings.warn(
+                "DataCollatorForWholeWordMask is only suitable for BertTokenizer-like tokenizers."
+                "Please refer to the documentation for more information."
+            )
 
         cand_indexes = []
         for (i, token) in enumerate(input_tokens):


Test patch: