Repository: huggingface/transformers
Pull request: Fix missing post_processor in DebertaV2Tokenizer causing no special t…

  # What does this PR do?                                                                                                                                                                
                                                                                                                                                                                           
  In transformers v5, `DebertaV2Tokenizer` was rewritten to use `TokenizersBackend`, but the `post_processor` responsible for adding `[CLS]`/`[SEP]` tokens was never set. This causes     
  `add_special_tokens=True` to silently produce output without special tokens for models like `microsoft/mdeberta-v3-base`.                                                                
                                                                                                                                                                                           
  The root cause: in v4, special tokens were added via `build_inputs_with_special_tokens()` (Python-level). In v5, this method was removed in favor of the Rust-level `post_processor` on  
  the `tokenizers` backend — but for `DebertaV2Tokenizer` this processor was never configured. Other tokenizers like `BertTokenizer` set it correctly.                                     
                                                                                                                                                                                           
  The fix adds a `TemplateProcessing` post-processor after `super().__init__()`, matching the pattern used by `BertTokenizer` and the template previously defined in                       
  `DebertaV2Converter.post_processor()`.                                                                                                                                                   

  Fixes #44568

  ## Before submitting
  - [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  - [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
        Pull Request section?
  - [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
        to it if that's the case. #44568
  - [ ] Did you make sure to update the documentation with your changes? Here are the
        [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
        [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
  - [x] Did you write any new necessary tests?

  ## Who can review?

  @ArthurZucker @itazap — tokenizers

Changed files:
filename: src/transformers/models/deberta_v2/tokenization_deberta_v2.py
patch: @@ -13,7 +13,7 @@
 # limitations under the License.
 """Tokenization class for model DeBERTa-v2."""
 
-from tokenizers import Regex, Tokenizer, decoders, normalizers, pre_tokenizers
+from tokenizers import Regex, Tokenizer, decoders, normalizers, pre_tokenizers, processors
 from tokenizers.models import Unigram
 
 from ...tokenization_utils_tokenizers import TokenizersBackend
@@ -153,5 +153,17 @@ def __init__(
             **kwargs,
         )
 
+        cls_token_id = self.cls_token_id if self.cls_token_id is not None else 0
+        sep_token_id = self.sep_token_id if self.sep_token_id is not None else 0
+
+        self._tokenizer.post_processor = processors.TemplateProcessing(
+            single=f"{str(self.cls_token)}:0 $A:0 {str(self.sep_token)}:0",
+            pair=f"{str(self.cls_token)}:0 $A:0 {str(self.sep_token)}:0 $B:1 {str(self.sep_token)}:1",
+            special_tokens=[
+                (str(self.cls_token), cls_token_id),
+                (str(self.sep_token), sep_token_id),
+            ],
+        )
+
 
 __all__ = ["DebertaV2Tokenizer"]

filename: tests/models/deberta_v2/test_tokenization_deberta_v2.py
patch: @@ -117,3 +117,20 @@ def test_do_lower_case_false_split_by_punct_false(self):
         tokens = tokenizer.convert_ids_to_tokens(tokenizer.encode(sequence, add_special_tokens=False))
 
         self.assertListEqual(tokens, tokens_target)
+
+    def test_post_processor_adds_special_tokens(self):
+        extractor = SentencePieceExtractor(SAMPLE_VOCAB)
+        vocab, vocab_scores, merges = extractor.extract()
+        tokenizer = DebertaV2Tokenizer(vocab=vocab_scores, unk_token="<unk>")
+
+        encoding = tokenizer("Hello world")
+        tokens = tokenizer.convert_ids_to_tokens(encoding["input_ids"])
+        self.assertEqual(tokens[0], "[CLS]")
+        self.assertEqual(tokens[-1], "[SEP]")
+
+        encoding_pair = tokenizer("Hello", "World")
+        tokens_pair = tokenizer.convert_ids_to_tokens(encoding_pair["input_ids"])
+        self.assertEqual(tokens_pair[0], "[CLS]")
+        sep_indices = [i for i, t in enumerate(tokens_pair) if t == "[SEP]"]
+        self.assertEqual(len(sep_indices), 2)
+        self.assertEqual(sep_indices[-1], len(tokens_pair) - 1)