Repository: keras-team/keras-nlp

Issue:
Reconcile example style in BPE docstring
The current BPE docstring ([link](https://github.com/keras-team/keras-nlp/blob/9aa8ac59e15814e975c3293e30af1f1b3d9aea32/keras_nlp/tokenizers/byte_pair_tokenizer.py#L188)) does not match the style of others in our library (e.g., [link](https://github.com/keras-team/keras-nlp/blob/9aa8ac59e15814e975c3293e30af1f1b3d9aea32/keras_nlp/models/roberta/roberta_preprocessing.py#L66)).
* Use three-backticks-with-python style to fence the blocks
* Do not reference external files outside of the `from_preset` constructor (which doesn't apply here)
* Do not show output of code


Discussion:
Command line-style with printed output is commonly used in KerasNLP and Keras, e.g., https://github.com/keras-team/keras-nlp/blob/master/keras_nlp/tokenizers/word_piece_tokenizer.py#L223, it has the advantage for easy docstring testing.  
Thanks for the comments @chenmoneygithub. It seems like we need to find a unified style because our examples in the `models` section look totally different.

Regardless of style, we do not want raw references to our GCP files since that's not the user journey.
Agreed re avoiding listing the GCP files!

Re docs styles. Tensorflow, Keras, and KerasNLP actually support two forms of doc string code:

 - `>>>` style (this is [python doctest](https://docs.python.org/3/library/doctest.html)). With explicit, tested output. We use this for our raw tokenizers, and generally when code samples are short and output is fixed.
 - ``  ``` `` style (this is called "fenced" code blocks). These are not tested, but we would like to add this https://github.com/keras-team/keras-nlp/issues/420.

If we add testing for fenced blocks, we could consider using that style everywhere for KerasNLP. Though it is not very compact for rapid short examples.

I suspect core Keras will continue to use both styles (and adding testing for all "fenced" blocks for core Keras would be hard).

Resolving patch:
diff --git a/keras_nlp/tokenizers/byte_pair_tokenizer.py b/keras_nlp/tokenizers/byte_pair_tokenizer.py
--- a/keras_nlp/tokenizers/byte_pair_tokenizer.py
+++ b/keras_nlp/tokenizers/byte_pair_tokenizer.py
@@ -187,8 +187,7 @@ class BytePairTokenizer(tokenizer.Tokenizer):
 
     Examples:
 
-    Use in-memory vocabulary and merge list.
-
+    Tokenize
     >>> vocab = {"butter": 1, "fly": 2}
     >>> merge = ["b u", "t t", "e r", "bu tt", "butt er", "f l", "fl y"]
     >>> tokenizer = keras_nlp.tokenizers.BytePairTokenizer(vocab, merge)
@@ -205,23 +204,6 @@ class BytePairTokenizer(tokenizer.Tokenizer):
     array([[1, 2],
            [1, 0]], dtype=int32)>
 
-    Use hosted vocabluary and merge list.
-
-    ```python
-    vocab_path = tf.keras.utils.get_file(
-        "vocab.json",
-        "https://storage.googleapis.com/keras-nlp/models/roberta_base/vocab.json",
-    )
-    merge_path = tf.keras.utils.get_file(
-        "merges.txt",
-        "https://storage.googleapis.com/keras-nlp/models/roberta_base/merges.txt",
-    )
-    tokenizer = BytePairTokenizer(
-        vocabulary=vocab_path, merges=merge_path
-    )
-    tokenizer("Butterfly is not flying butter!")
-    ```
-
     Detokenize
     >>> vocab = {"butter": 1, "fly": 2}
     >>> merge = ["b u", "t t", "e r", "bu tt", "butt er", "f l", "fl y"]
@@ -229,7 +211,6 @@ class BytePairTokenizer(tokenizer.Tokenizer):
     >>> tokenizer.detokenize([[1, 2]])
     <tf.Tensor: shape=(1,), dtype=string, numpy=array([b'butterfly'],
     dtype=object)>
-
     """
 
     def __init__(


Test patch: