Repository: keras-team/keras-nlp

Issue:
Deberta tokenizer.detokenize() errors out with mask token
When working on the Deberta masked language model, we had to do some special treatment for the mask token in the tokenizer.

We left one outstanding bug on the main PR, which is that detokenize will error out with a mask token. See:
https://github.com/keras-team/keras-nlp/pull/732#issuecomment-1449746110

Here's a colab:
https://colab.research.google.com/gist/mattdangerw/5164a7cad80e9f5fcbb9a495264f80e1/deberta-detokenize-error.ipynb

We should either strip or properly render the mask token during detokenize so the call does not error out.


Discussion:
@abheesht17 copying you in case you have any thoughts on a proper fix.
/assign

Hey @mattdangerw can I work this issue ?

sg!
So, here it is two ways to tackle this error, strip the mask token or properly rendering during detokenize while calling 🤔
So, an approach I have in mind is that:
In the [id_to_token](https://github.com/keras-team/keras-nlp/blob/4c94a0d68872a72ed3e27d3f93a3d7eec3ddc0ba/keras_nlp/tokenizers/sentence_piece_tokenizer.py#L170) and [token_to_id](https://github.com/keras-team/keras-nlp/blob/4c94a0d68872a72ed3e27d3f93a3d7eec3ddc0ba/keras_nlp/tokenizers/sentence_piece_tokenizer.py#L174), 
we can just use an if-else loop to check if the id is equal to the self.mask_token_id and token is equal to [MASK] and return accordingly. Does that work?
I believe it's a typo as there is no change to the BBPE tokenizer used in DeBerta (earlier versions). Correct me if I'm wrong, thanks

https://github.com/keras-team/keras-nlp/blob/4c94a0d68872a72ed3e27d3f93a3d7eec3ddc0ba/keras_nlp/models/deberta_v3/deberta_v3_masked_lm_preprocessor.py#L141
> So, an approach I have in mind is that: In the [id_to_token](https://github.com/keras-team/keras-nlp/blob/4c94a0d68872a72ed3e27d3f93a3d7eec3ddc0ba/keras_nlp/tokenizers/sentence_piece_tokenizer.py#L170) and [token_to_id](https://github.com/keras-team/keras-nlp/blob/4c94a0d68872a72ed3e27d3f93a3d7eec3ddc0ba/keras_nlp/tokenizers/sentence_piece_tokenizer.py#L174), we can just use an if-else loop to check if the id is equal to the self.mask_token_id and token is equal to [MASK] and return accordingly. Does that work?

The issue is with the `detokenize` function. We have already taken care of the functions you have mentioned: https://github.com/keras-team/keras-nlp/blob/master/keras_nlp/models/deberta_v3/deberta_v3_tokenizer.py#L113-L121.
> So, here it is two ways to tackle this error, strip the mask token or properly rendering during detokenize while calling 🤔

The latter approach seems more intuitive to me. Stripping the mask token means that we ignore `[MASK]` during detokenization?   
> > So, here it is two ways to tackle this error, strip the mask token or properly rendering during detokenize while calling 🤔
> 
> The latter approach seems more intuitive to me. Stripping the mask token means that we ignore `[MASK]` during detokenization?

not sure but stripping removes or truncates extra spaces ,
So, what about rendering the mask token during detokenization ?
> not sure but stripping removes or truncates extra spaces , So, what about rendering the mask token during detokenization ?

I misunderstood what you meant, sorry. In my mind, I think @TheAthleticCoder had the right idea, you could override `detokenize` and use `id_to_token` for each input, which already takes care of `[MASK]`.

There might be a more efficient method though.
I would suggest that we just strip the mask token for now. @ADITYADAS1999, could you please open a PR for that?

Trying to keep the mask token will most probably require modifying the sentencepiece proto. If anyone is willing to take that up, cool! Here is the repo: https://github.com/google/sentencepiece. Here is a notebook to get started: https://colab.research.google.com/github/google/sentencepiece/blob/master/python/sentencepiece_python_module_example.ipynb. This isn't very easy to do since the SP Python API does not have any way to add tokens after training :/

Resolving patch:
diff --git a/keras_nlp/models/deberta_v3/deberta_v3_tokenizer.py b/keras_nlp/models/deberta_v3/deberta_v3_tokenizer.py
--- a/keras_nlp/models/deberta_v3/deberta_v3_tokenizer.py
+++ b/keras_nlp/models/deberta_v3/deberta_v3_tokenizer.py
@@ -16,6 +16,8 @@
 
 import copy
 
+import tensorflow as tf
+
 from keras_nlp.api_export import keras_nlp_export
 from keras_nlp.models.deberta_v3.deberta_v3_presets import backbone_presets
 from keras_nlp.tokenizers.sentence_piece_tokenizer import SentencePieceTokenizer
@@ -120,6 +122,10 @@ def token_to_id(self, token):
             return self.mask_token_id
         return super().token_to_id(token)
 
+    def detokenize(self, ids):
+        ids = tf.ragged.boolean_mask(ids, tf.not_equal(ids, self.mask_token_id))
+        return super().detokenize(ids)
+
     @classproperty
     def presets(cls):
         return copy.deepcopy(backbone_presets)


Test patch:
diff --git a/keras_nlp/models/deberta_v3/deberta_v3_tokenizer_test.py b/keras_nlp/models/deberta_v3/deberta_v3_tokenizer_test.py
--- a/keras_nlp/models/deberta_v3/deberta_v3_tokenizer_test.py
+++ b/keras_nlp/models/deberta_v3/deberta_v3_tokenizer_test.py
@@ -64,6 +64,11 @@ def test_detokenize(self):
         output = self.tokenizer.detokenize(input_data)
         self.assertEqual(output, tf.constant(["the quick brown fox"]))
 
+    def test_detokenize_mask_token(self):
+        input_data = tf.constant([[4, 9, 5, 7, self.tokenizer.mask_token_id]])
+        output = self.tokenizer.detokenize(input_data)
+        self.assertEqual(output, tf.constant(["the quick brown fox"]))
+
     def test_vocabulary_size(self):
         self.assertEqual(self.tokenizer.vocabulary_size(), 11)