Repository: keras-team/keras-nlp

Issue:
Add a utility to detokenize as lists of python strings
Going from tf.Tensor strings back to python string is confusing. We should add a utility to do that to the base tokenizer class.

We can discuss the exact name of the PR, suggest `.detokenize_to_strings()` as a starting point.

Usages:
```
>>> keras_nlp.layers.ByteTokenizer().detokenize_to_strings([104, 101, 108, 108, 111])
'hello'
>>> keras_nlp.layers.ByteTokenizer().detokenize_to_strings([[104, 101, 108, 108, 111]])
['hello']
```




Discussion:
@aflah02 can you comment anything on this issue? I can't assign you until you are a participant :)
I think the flow for this utility will need to be something like...

For ragged tensors, call `x.to_list()`. For dense tensors, call `x.numpy().tolist()`. Then recursively walk all elements and convert byte strings to python strings.
Thanks for creating this, I'll get to work on this!
@mattdangerw I think I've got it down roughly
You can see the demo [here](https://colab.research.google.com/gist/aflah02/df745134ff33053334653e7d8992108b/detokenizing.ipynb)
The only change was to add a method in the Base Class which is implemented there itself and used by all Classes which Inherit From it Right? Here's the code for the [same](https://github.com/aflah02/keras-nlp/blob/b98837ef4a1d349baf646294395fa5dca920f552/keras_nlp/tokenizers/tokenizer.py#L133)
If this works I'll write some tests for the same
That looks good!

Yeah, we should only need to add this to the base class. I'm still not 100% sure about the name, but let's make a PR and we can discuss there.

Things to test:
 - ragged ouputs
 - dense outputs
 - scalar dense outputs (outputting a single string)

We should have a unit test for each of those cases.

Resolving patch:
diff --git a/keras_nlp/__init__.py b/keras_nlp/__init__.py
--- a/keras_nlp/__init__.py
+++ b/keras_nlp/__init__.py
@@ -15,5 +15,6 @@
 from keras_nlp import layers
 from keras_nlp import metrics
 from keras_nlp import tokenizers
+from keras_nlp import utils
 
 __version__ = "0.1.1"
diff --git a/keras_nlp/utils/tensor_utils.py b/keras_nlp/utils/tensor_utils.py
new file mode 100644
--- /dev/null
+++ b/keras_nlp/utils/tensor_utils.py
@@ -0,0 +1,47 @@
+# Copyright 2022 The KerasNLP Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import tensorflow as tf
+
+
+def _decode_strings_to_utf8(inputs):
+    """Recursively decodes to list of strings with 'utf-8' encoding."""
+    if isinstance(inputs, bytes):
+        # Handles the case when the input is a scalar string.
+        return inputs.decode("utf-8")
+    else:
+        # Recursively iterate when input is a list.
+        return [_decode_strings_to_utf8(x) for x in inputs]
+
+
+def tensor_to_string_list(inputs):
+    """Detokenize and convert tensor to nested lists of python strings.
+
+    This is a convenience method which converts each byte string to a python
+    string.
+
+    Args:
+        inputs: Input tensor, or dict/list/tuple of input tensors.
+        *args: Additional positional arguments.
+        **kwargs: Additional keyword arguments.
+    """
+    if not isinstance(inputs, (tf.RaggedTensor, tf.Tensor)):
+        inputs = tf.convert_to_tensor(inputs)
+    if isinstance(inputs, tf.RaggedTensor):
+        list_outputs = inputs.to_list()
+    elif isinstance(inputs, tf.Tensor):
+        list_outputs = inputs.numpy()
+        if inputs.shape.rank != 0:
+            list_outputs = list_outputs.tolist()
+    return _decode_strings_to_utf8(list_outputs)


Test patch:
diff --git a/keras_nlp/utils/tensor_utils_test.py b/keras_nlp/utils/tensor_utils_test.py
new file mode 100644
--- /dev/null
+++ b/keras_nlp/utils/tensor_utils_test.py
@@ -0,0 +1,33 @@
+# Copyright 2022 The KerasNLP Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import tensorflow as tf
+from tensor_utils import tensor_to_string_list
+
+
+class TensorToStringListTest(tf.test.TestCase):
+    def test_detokenize_to_strings_for_ragged(self):
+        input_data = tf.ragged.constant([["▀▁▂▃", "samurai"]])
+        detokenize_output = tensor_to_string_list(input_data)
+        self.assertAllEqual(detokenize_output, [["▀▁▂▃", "samurai"]])
+
+    def test_detokenize_to_strings_for_dense(self):
+        input_data = tf.constant([["▀▁▂▃", "samurai"]])
+        detokenize_output = tensor_to_string_list(input_data)
+        self.assertAllEqual(detokenize_output, [["▀▁▂▃", "samurai"]])
+
+    def test_detokenize_to_strings_for_scalar(self):
+        input_data = tf.constant("▀▁▂▃")
+        detokenize_output = tensor_to_string_list(input_data)
+        self.assertEqual(detokenize_output, "▀▁▂▃")