Extend `tokenizer.pad` with `offset_mapping`
Author: vadimirtlach
Pull request: yes

# What does this PR do?

Extend `tokenizer.pad`, accurately `tokenizer._pad`, method with functionality for padding `offset_mapping` with `(0, 0)`.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#start-contributing-pull-requests),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [Padding offsets mapping via tokenizer.pad #18681
](https://github.com/huggingface/transformers/issues/18681)? Please add a link
      to it if that's the case.
- [] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @LysandreJik

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @n1t0, @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_18705). All of your documentation changes will be reflected on that endpoint.

Hi @vad13irt. Could you explain a bit why this change is needed, and what would be its usage? With a short code snippet might be very helpful. Thank you!





> Hi @vad13irt. Could you explain a bit why this change is needed, and what would be its usage? With a short code snippet might be very helpful. Thank you!

Hello, @ydshieh! Thank you for your question.

Why is it needed? 

Sometimes, there is a need in converting token-wise predictions to char-wise predictions, because you shoud create the zero array with the length of input text, iterate over all tokens and their corresponding spans and then change indices in the zero-array to get char-wise predictions; or compute confidence scores of token-wise predictions and then selecting the most confident candidates, and then compute some losses or metrics. 


For example, it is done in this [great repository](https://github.com/affjljoo3581/Feedback-Prize-Competition/blob/master/src/lightning/module.py) for one of the Kaggle competitions. 


Let's take a look:


```py

    def validation_epoch_end(self, outputs: List[Tuple[torch.Tensor, ...]]):
        loss, texts, logits, labels, offset_mapping = map(list, zip(*outputs))
        texts = sum(texts, [])

        # Decode the NER-tags with beam-search algorithm.
        preds, pred_probs = ner_beam_search_decode(
            concat_tensors_with_padding(logits, padding=0).float().log_softmax(dim=-1),
            self.id2label,
            self.config.model.decoding.beam_size,
        )
        labels = concat_tensors_with_padding(labels, padding=-100)
        offset_mapping = concat_tensors_with_padding(offset_mapping, padding=0)

        preds, pred_probs = preds.cpu().numpy(), pred_probs.cpu().numpy()
        labels, offset_mapping = labels.cpu().numpy(), offset_mapping.cpu().numpy()

        # Collect the NER entities for predictions and labels to calculate the F1 score.
        pred_entities_list, label_entities_list = [], []
        for text, preds, pred_probs, labels, offset_mapping in zip(
            texts, preds, pred_probs, labels, offset_mapping
        ):
            valid_mask = offset_mapping[..., 1] > 0

            preds, pred_probs = preds[valid_mask], pred_probs[valid_mask]
            labels, offset_mapping = labels[valid_mask], offset_mapping[valid_mask]

            # Extract the NER entities from BIO-naming tags. Note that the
            # low-confidence or too-short entities will be dropped.
            pred_entities, pred_entity_probs = extract_entities_from_ner_tags(
                [self.id2label[x] for x in preds], offset_mapping, pred_probs
            )
            pred_entities = convert_offsets_to_word_indices(text, pred_entities)

            pred_entities = [
                (entity, a, b)
                for (entity, a, b), prob in zip(pred_entities, pred_entity_probs)
                if b - a + 1 >= self.minimum_lengths[entity]
                and prob >= self.minimum_probs[entity]
            ]
            pred_entities_list.append(pred_entities)

            # Of course, we will extract the entities for labels.
            label_entities, _ = extract_entities_from_ner_tags(
                [self.id2label[x] for x in labels], offset_mapping
            )
            label_entities = convert_offsets_to_word_indices(text, label_entities)
            label_entities_list.append(label_entities)

        # Calculate the macro-F1 score for NER entities.
        f1 = ner_entity_macro_f1_score(pred_entities_list, label_entities_list)

        self.log("val/loss", torch.stack(loss).mean())
        self.log("val/f1_score", f1)
```


In order to get the matrix representation of `offset_mapping` the user should pad them with this line of code, thereby alleviating the writing code for getting non-special tokens mask, although such behavior can be reached using `special_tokens_mask`. 

```py
        offset_mapping = concat_tensors_with_padding(offset_mapping, padding=0)
```

Then, he took the non-special tokens to further get the predicted entities and their confidence scores, e.g mean of char-wise predictions (please refer to this [code snippet](https://github.com/affjljoo3581/Feedback-Prize-Competition/blob/034427117cc8a3e1dd63401b3519fc28e3f18830/src/utils/ner_utils.py#L31)). The entities are selected with corresponding thresholds. Then, he computed F1-macro. 


If you have some questions, please tag me (@vad13irt). 

cc @SaulLu  :)

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.