Repository: keras-team/keras-nlp

Issue:
Add `from_preset` constructor to `BertPreprocessor`
# Proposal

In #387 we allowed construction of a BERT model from a "preset" model architecture and weights; for example `Bert.from_preset("bert_base_uncased_en")`. I propose to do the same with `BertPreprocessor`, automatically generating the matching preprocessing functionality using the same string id: `BertPreprocessor.from_preset("bert_base_uncased_en")`.

# Preset configuration

We will need to add preprocessing metadata to each preset id. One way to do this is keeping the vocabulary information normalized in a separate struct and joining it at call time. However, we could follow the pattern in #387 more closely by inlining the preprocessing config and weights:

```python
backbone_presets = {
    "bert_tiny_uncased_en": {
        "config": {
            "vocabulary_size": 30522,
            "num_layers": 2,
            "num_heads": 2,
            "hidden_dim": 128,
            "intermediate_dim": 512,
            "dropout": 0.1,
            "max_sequence_length": 512,
            "num_segments": 2,
        },
        "description": (
            "Tiny size of BERT where all input is lowercased. "
            "Trained on English Wikipedia + BooksCorpus."
        ),
        "weights_url": "https://storage.googleapis.com/keras-nlp/models/bert_tiny_uncased_en/model.h5",
        "weights_hash": "c2b29fcbf8f814a0812e4ab89ef5c068",
        "vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_base_uncased_en/vocab.txt",
        "vocabulary_hash": "64800d5d8528ce344256daf115d4965e",
        "lowercase": True,
    },
}
```

If we add more preprocessor arguments than `lowercase` in the future then a separate `preprocessor_config` field could be added. However, we have already hardcoded most preprocessing metadata in the `BertPreprocessor` class. Additional information is generally added in files like `vocab.txt` and `merges.txt` rather than flat parameters, so this doesn't seem helpful at this time.

# Signature
The arguments `sequence_length` and `truncate` are not determined by the preset alone and should be added to the signature. We must enforce that `sequence_length` is no longer than the training time setting however.

One complication is that most users will want to use the biggest sequence length that the model allows. We might want to set the default to `None` so that we can fill in this value unless the user has a different preference.

```python
class BertPreprocessor(keras.layers.Layer):
    # Generic constructor
    def __init__(
        self,
        vocabulary="uncased_en",
        lowercase=False,
        sequence_length=512,
        truncate="round_robin",
        **kwargs,
    ):
        pass

    @classmethod
    def from_preset(
        cls, 
        preset,
        sequence_length=512, 
        truncate="round_robin",
        **kwargs,
    ):
    if sequence_length > backbone_presets[preset]["config"]["max_sequence_length"]:
        raise ValueError(...)
```

# Preset namespaces
In the long term we will likely add task specific presets as well as backbone presets. Unlike model classes, Preprocessors could pertain to any preset type. Therefore long term we will have to create a single namespace of presets or explicitly check multiple namespaces and enforce uniqueness between them. 

Discussion:
jbischof: @fchollet, @mattdangerw is working on a `pipeline` class that composes preprocessing and the model. It would have its own `from_preset` method. Looking forward to discussing when we have a design doc.
fchollet: > @fchollet, @mattdangerw is working on a pipeline class that composes preprocessing and the model

I am not sure that a separate "pipeline" class is warranted. I'd favor including the preprocessing in the user-facing model (e.g. Bert or BertClassifier), similar to what we do for vision models.

- Models are instantiated with or without preprocessing included (defaults to True, i.e. with).
- With `fit()`, `predict()`, and `evaluate()`: the preprocessing is automatically mapped into the dataset (if it is included in the model).
- With `__call__()`: preprocessing is applied by default (if it is included in the model) but you have an argument to turn it off, which can be useful in various situations when writing low-level code.
jbischof: @fchollet I was assuming that regardless of how we implement preprocessing as part of the model we still need separate preprocessing to be a good experience, which I have attempted in this PR. Are you saying you want to block this PR until we figure out joint preprocessing or that the current PR is a bad experience for separate preprocessing?
mattdangerw: @fchollet and @jbischof, re how to access the pre-preprocessing paired with a model, I think we can do that as a follow up. (Basically I'm just echoing Jonathan's point above).

IMO, we should have a way to access preprocessing standalone--that's a totally valid user journey. This PR address that in a very consistent way with the UX we just came up with for `keras_nlp.models.Bert`.

As Francois was saying in the original question, the question is whether to *also* expose the preprocessing on the model. I think that is a fairly dense topic, and one that can roughly be summarized as "what do we want to do about pipelines"? Let's tackle that in a future PR!
fchollet: > Are you saying you want to block this PR until we figure out joint preprocessing or that the current PR is a bad experience for separate preprocessing?

No, it's clear that being able to retrieve the tokenizer independently is useful. I was saying:

> should they be **also** be available from the corresponding name model
chenmoneygithub: I tried this implementation and spotted a bug:
```
bert_preprocessor = keras_nlp.models.BertPreprocessor.from_preset("bert_small_uncased_en")
inputs = ["This is a testing string!", "Another test string!"]
inputs = bert_preprocessor(inputs)
print(inputs)
```

This gives shape (512, ) for `token_ids`, `segment_ids` and `padding_mask`, which means the batch information is lost. 
jbischof: @chenmoneygithub the preprocessing layer does not handle batch inputs. It is expected to be mapped over each example in the batch. If there are multiple sentences in the input, these are interpreted as multiple segments for the same example (e.g., for NSP or sentence similarity).
mattdangerw: @jbischof @chenmoneygithub I think this batching convo is unrelated to the PR. But seems worth clearing up.

 - All our preprocessing does handle batched and unbatched input. Loading a dataset with, say, tfds batched, and piping it through our preprocessing is actually a pretty core use case! (And we do support it universally)
 - All our preprocessing supports converting to tensors, just as a convenience. (Most tf ops do as well)
 - The `BertPreprocessor` supports multiple (batched or unbatched) inputs which are then collated with `sep` tokens. This is also the behavior of the `MultiSegmentPacker` layer.

The sum total of these means you need to be careful when passing to un-tensorized inputs to the layer, because you hit an ambiguity of sorts. We treat the first tuple or list we see as indicating multiple separate string features to collate. So in the code snippet you posted Chen, we are doing a very valid thing, and contacting the two sentences together in a single sequence. This colab shows what I mean.
https://gist.github.com/mattdangerw/61b74938a11b61f1ce858b770c601bd6

We could consider flipping the default behavior there, so if you pass a tuple of strings (rather than a tuple of numpy or tensors) we treat this as a batch. That is definitely something for a separate issue though!

Patch:
diff --git a/keras_nlp/models/bert/bert_models.py b/keras_nlp/models/bert/bert_models.py
index 0fb2b16b..654f050c 100644
--- a/keras_nlp/models/bert/bert_models.py
+++ b/keras_nlp/models/bert/bert_models.py
@@ -23,18 +23,13 @@ from tensorflow import keras
 from keras_nlp.layers.position_embedding import PositionEmbedding
 from keras_nlp.layers.transformer_encoder import TransformerEncoder
 from keras_nlp.models.bert.bert_presets import backbone_presets
+from keras_nlp.models.utils import classproperty
 
 
 def bert_kernel_initializer(stddev=0.02):
     return keras.initializers.TruncatedNormal(stddev=stddev)
 
 
-# TODO(jbischof): refactor into util file
-class classproperty(property):
-    def __get__(self, _, owner_cls):
-        return self.fget(owner_cls)
-
-
 @keras.utils.register_keras_serializable(package="keras_nlp")
 class Bert(keras.Model):
     """BERT encoder network.
@@ -289,5 +284,5 @@ FROM_PRESET_DOCSTRING = """Instantiate BERT model from preset architecture and w
 setattr(
     Bert.from_preset.__func__,
     "__doc__",
-    FROM_PRESET_DOCSTRING.format(names=", ".join(backbone_presets)),
+    FROM_PRESET_DOCSTRING.format(names=", ".join(Bert.presets)),
 )
diff --git a/keras_nlp/models/bert/bert_preprocessing.py b/keras_nlp/models/bert/bert_preprocessing.py
index ae461bab..5c06977f 100644
--- a/keras_nlp/models/bert/bert_preprocessing.py
+++ b/keras_nlp/models/bert/bert_preprocessing.py
@@ -13,113 +13,95 @@
 # limitations under the License.
 """BERT preprocessing layers."""
 
+import copy
 import os
 
 from tensorflow import keras
 
 from keras_nlp.layers.multi_segment_packer import MultiSegmentPacker
-from keras_nlp.models.bert.bert_presets import vocabularies
+from keras_nlp.models.bert.bert_presets import backbone_presets
+from keras_nlp.models.utils import classproperty
 from keras_nlp.tokenizers.word_piece_tokenizer import WordPieceTokenizer
 
 
-def _handle_pretrained_tokenizer_arguments(vocabulary, lowercase):
-    """Look up pretrained defaults for tokenizer arguments.
-
-    This helper will validate the `vocabulary` and `lowercase` arguments, and
-    fully resolve them in the case we are loading pretrained weights.
-    """
-
-    # TODO(jbischof): allow user to lookup vocabulary from checkpoint name
-    if isinstance(vocabulary, str) and vocabulary in vocabularies:
-        metadata = vocabularies[vocabulary]
-        vocabulary = keras.utils.get_file(
-            "vocab.txt",
-            metadata["vocabulary_url"],
-            cache_subdir=os.path.join("models", "bert", vocabulary),
-            file_hash=metadata["vocabulary_hash"],
+class BertPreprocessor(keras.layers.Layer):
+    """BERT preprocessing layer.
+
+    This preprocessing layer will do three things:
+
+     - Tokenize any number of inputs using a
+       `keras_nlp.tokenizers.WordPieceTokenizer`.
+     - Pack the inputs together using a `keras_nlp.layers.MultiSegmentPacker`.
+       with the appropriate `"[CLS]"`, `"[SEP]"` and `"[PAD]"` tokens.
+     - Construct a dictionary of with keys `"token_ids"`, `"segment_ids"`,
+       `"padding_mask"`, that can be passed directly to a BERT model.
+
+    This layer will accept either a tuple of (possibly batched) inputs, or a
+    single input tensor. If a single tensor is passed, it will be packed
+    equivalently to a tuple with a single element.
+
+    The `WordPieceTokenizer` can be accessed via the `tokenizer` property on
+    this layer, and can be used directly for custom packing on inputs.
+
+    Args:
+        vocabulary: A list of vocabulary terms or a vocabulary filename.
+        lowercase: If `True`, input will be lowercase before tokenization. If
+            `vocabulary` is set to a pretrained vocabulary, this parameter will
+            be inferred.
+        sequence_length: The length of the packed inputs.
+        truncate: string. The algorithm to truncate a list of batched segments
+            to fit within `sequence_length`. The value can be either
+            `round_robin` or `waterfall`:
+                - `"round_robin"`: Available space is assigned one token at a
+                    time in a round-robin fashion to the inputs that still need
+                    some, until the limit is reached.
+                - `"waterfall"`: The allocation of the budget is done using a
+                    "waterfall" algorithm that allocates quota in a
+                    left-to-right manner and fills up the buckets until we run
+                    out of budget. It supports an arbitrary number of segments.
+
+    Examples:
+    ```python
+    vocab = ["[UNK]", "the", "qu", "##ick", "br", "##own", "fox", "tripped"]
+    vocab += ["call", "me", "ish", "##mael", "."]
+    vocab += ["oh", "look", "a", "whale"]
+    vocab += ["i", "forgot", "my", "home", "##work"]
+    preprocessor = keras_nlp.models.BertPreprocessor(vocabulary=vocab)
+
+    # Tokenize and pack a single sentence directly.
+    preprocessor("The quick brown fox jumped.")
+
+    # Tokenize and pack a multiple sentence directly.
+    preprocessor(("The quick brown fox jumped.", "Call me Ishmael."))
+
+    # Map a dataset to preprocess a single sentence.
+    features = ["The quick brown fox jumped.", "I forgot my homework."]
+    labels = [0, 1]
+    ds = tf.data.Dataset.from_tensor_slices((features, labels))
+    ds = ds.map(
+        lambda x, y: (preprocessor(x), y),
+        num_parallel_calls=tf.data.AUTOTUNE,
+    )
+
+    # Map a dataset to preprocess a multiple sentences.
+    first_sentences = ["The quick brown fox jumped.", "Call me Ishmael."]
+    second_sentences = ["The fox tripped.", "Oh look, a whale."]
+    labels = [1, 1]
+    ds = tf.data.Dataset.from_tensor_slices(
+        (
+            (first_sentences, second_sentences), labels
         )
-        lowercase = metadata["lowercase"]
-
-    return vocabulary, lowercase
-
-
-PREPROCESSOR_DOCSTRING = """BERT preprocessor with pretrained vocabularies.
-
-This preprocessing layer will do three things:
-
- - Tokenize any number of inputs using a
-   `keras_nlp.tokenizers.WordPieceTokenizer`.
- - Pack the inputs together using a `keras_nlp.layers.MultiSegmentPacker`.
-   with the appropriate `"[CLS]"`, `"[SEP]"` and `"[PAD]"` tokens.
- - Construct a dictionary of with keys `"token_ids"`, `"segment_ids"`,
-   `"padding_mask"`, that can be passed directly to a BERT model.
-
-This layer will accept either a tuple of (possibly batched) inputs, or a single
-input tensor. If a single tensor is passed, it will be packed equivalently to
-a tuple with a single element.
-
-The WordPiece tokenizer can be accessed via the `tokenizer` property on this
-layer, and can be used directly for custom packing on inputs.
-
-Args:
-    vocabulary: One of a list of vocabulary terms, a vocabulary filename, or
-        the name of the pretrained vocabulary. For a pretrained vocabulary,
-        `vocabulary` should be one of {names}, and should match the `weights`
-        parameter of any pretrained BERT model.
-    lowercase: If `True`, input will be lowercase before tokenization. If
-        `vocabulary` is set to a pretrained vocabulary, this parameter will
-        be inferred.
-    sequence_length: The length of the packed inputs. Only used if
-        `pack_inputs` is True.
-    truncate: The algorithm to truncate a list of batched segments to fit
-        within `sequence_length`. Only used if
-        `pack_inputs` is True. The value can be either `round_robin` or
-        `waterfall`:
-            - `"round_robin"`: Available space is assigned one token at a
-                time in a round-robin fashion to the inputs that still need
-                some, until the limit is reached.
-            - `"waterfall"`: The allocation of the budget is done using a
-                "waterfall" algorithm that allocates quota in a
-                left-to-right manner and fills up the buckets until we run
-                out of budget. It supports an arbitrary number of segments.
-
-Examples:
-```python
-preprocessor = keras_nlp.models.BertPreprocessor(vocabulary="uncased_en")
-
-# Tokenize and pack a single sentence directly.
-preprocessor("The quick brown fox jumped.")
-
-# Tokenize and pack a multiple sentence directly.
-preprocessor(("The quick brown fox jumped.", "Call me Ishmael."))
-
-# Map a dataset to preprocess a single sentence.
-features = ["The quick brown fox jumped.", "I forgot my homework."]
-labels = [0, 1]
-ds = tf.data.Dataset.from_tensor_slices((features, labels))
-ds = ds.map(
-    lambda x, y: (preprocessor(x), y),
-    num_parallel_calls=tf.data.AUTOTUNE,
-)
-
-# Map a dataset to preprocess a multiple sentences.
-first_sentences = ["The quick brown fox jumped.", "Call me Ishmael."]
-second_sentences = ["The fox tripped.", "Oh look, a whale."]
-labels = [1, 1]
-ds = tf.data.Dataset.from_tensor_slices(((first_sentences, second_sentences), labels))
-ds = ds.map(
-    lambda x, y: (preprocessor(x), y),
-    num_parallel_calls=tf.data.AUTOTUNE,
-)
-```
-"""
-
+    )
+    ds = ds.map(
+        lambda x, y: (preprocessor(x), y),
+        num_parallel_calls=tf.data.AUTOTUNE,
+    )
+    ```
+    """
 
-# TODO(jbischof): add `from_preset` classmethod
-class BertPreprocessor(keras.layers.Layer):
     def __init__(
         self,
-        vocabulary="uncased_en",
+        vocabulary,
         lowercase=False,
         sequence_length=512,
         truncate="round_robin",
@@ -127,10 +109,6 @@ class BertPreprocessor(keras.layers.Layer):
     ):
         super().__init__(**kwargs)
 
-        vocabulary, lowercase = _handle_pretrained_tokenizer_arguments(
-            vocabulary, lowercase
-        )
-
         self.tokenizer = WordPieceTokenizer(
             vocabulary=vocabulary,
             lowercase=lowercase,
@@ -185,9 +163,93 @@ class BertPreprocessor(keras.layers.Layer):
             "padding_mask": token_ids != self.pad_token_id,
         }
 
+    @classproperty
+    def presets(cls):
+        return copy.deepcopy(backbone_presets)
+
+    @classmethod
+    def from_preset(
+        cls,
+        preset,
+        sequence_length=None,
+        truncate="round_robin",
+        **kwargs,
+    ):
+        if preset not in cls.presets:
+            raise ValueError(
+                "`preset` must be one of "
+                f"""{", ".join(cls.presets)}. Received: {preset}."""
+            )
+        metadata = cls.presets[preset]
+
+        vocabulary = keras.utils.get_file(
+            "vocab.txt",
+            metadata["vocabulary_url"],
+            cache_subdir=os.path.join("models", preset),
+            file_hash=metadata["vocabulary_hash"],
+        )
+
+        config = metadata["preprocessor_config"]
+        # Use model's `max_sequence_length` if `sequence_length` unspecified;
+        # otherwise check that `sequence_length` not too long.
+        max_sequence_length = metadata["config"]["max_sequence_length"]
+        if sequence_length is not None:
+            if sequence_length > max_sequence_length:
+                raise ValueError(
+                    f"`sequence_length` cannot be longer than `{preset}` "
+                    f"preset's `max_sequence_length` of {max_sequence_length}. "
+                    f"Received: {sequence_length}."
+                )
+        else:
+            sequence_length = max_sequence_length
+
+        config.update(
+            {
+                "sequence_length": sequence_length,
+                "vocabulary": vocabulary,
+                "truncate": truncate,
+            },
+        )
+
+        return cls.from_config({**config, **kwargs})
+
+
+FROM_PRESET_DOCSTRING = """Instantiate BERT preprocessor from preset architecture.
+
+    Args:
+        preset: string. Must be one of {names}.
+        sequence_length: int, optional. The length of the packed inputs. Must be
+            equal to or smaller than the `max_sequence_length` of the preset. If
+            left as default, the `max_sequence_length` of the preset will be
+            used.
+        truncate: string. The algorithm to truncate a list of batched segments
+            to fit within `sequence_length`. The value can be either
+            `round_robin` or `waterfall`:
+                - `"round_robin"`: Available space is assigned one token at a
+                    time in a round-robin fashion to the inputs that still need
+                    some, until the limit is reached.
+                - `"waterfall"`: The allocation of the budget is done using a
+                    "waterfall" algorithm that allocates quota in a
+                    left-to-right manner and fills up the buckets until we run
+                    out of budget. It supports an arbitrary number of segments.
+
+    Examples:
+    ```python
+    # Load preprocessor from preset
+    preprocessor = BertPreprocessor.from_preset("bert_base_uncased_en")
+    preprocessor("The quick brown fox jumped.")
+
+    # Override sequence_length
+    preprocessor = BertPreprocessor.from_preset(
+        "bert_base_uncased_en"
+        sequence_length=64
+    )
+    preprocessor("The quick brown fox jumped.")
+    ```
+    """
 
 setattr(
-    BertPreprocessor,
+    BertPreprocessor.from_preset.__func__,
     "__doc__",
-    PREPROCESSOR_DOCSTRING.format(names=", ".join(vocabularies)),
+    FROM_PRESET_DOCSTRING.format(names=", ".join(BertPreprocessor.presets)),
 )
diff --git a/keras_nlp/models/bert/bert_presets.py b/keras_nlp/models/bert/bert_presets.py
index 2f448cd5..728444ca 100644
--- a/keras_nlp/models/bert/bert_presets.py
+++ b/keras_nlp/models/bert/bert_presets.py
@@ -26,13 +26,17 @@ backbone_presets = {
             "max_sequence_length": 512,
             "num_segments": 2,
         },
-        "vocabulary": "uncased_en",
+        "preprocessor_config": {
+            "lowercase": True,
+        },
         "description": (
             "Tiny size of BERT where all input is lowercased. "
             "Trained on English Wikipedia + BooksCorpus."
         ),
         "weights_url": "https://storage.googleapis.com/keras-nlp/models/bert_tiny_uncased_en/model.h5",
         "weights_hash": "c2b29fcbf8f814a0812e4ab89ef5c068",
+        "vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_tiny_uncased_en/vocab.txt",
+        "vocabulary_hash": "64800d5d8528ce344256daf115d4965e",
     },
     "bert_small_uncased_en": {
         "config": {
@@ -45,13 +49,17 @@ backbone_presets = {
             "max_sequence_length": 512,
             "num_segments": 2,
         },
-        "vocabulary": "uncased_en",
+        "preprocessor_config": {
+            "lowercase": True,
+        },
         "description": (
             "Small size of BERT where all input is lowercased. "
             "Trained on English Wikipedia + BooksCorpus."
         ),
         "weights_url": "https://storage.googleapis.com/keras-nlp/models/bert_small_uncased_en/model.h5",
         "weights_hash": "08632c9479b034f342ba2c2b7afba5f7",
+        "vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_small_uncased_en/vocab.txt",
+        "vocabulary_hash": "64800d5d8528ce344256daf115d4965e",
     },
     "bert_medium_uncased_en": {
         "config": {
@@ -64,13 +72,17 @@ backbone_presets = {
             "max_sequence_length": 512,
             "num_segments": 2,
         },
-        "vocabulary": "uncased_en",
+        "preprocessor_config": {
+            "lowercase": True,
+        },
         "description": (
             "Medium size of BERT where all input is lowercased. "
             "Trained on English Wikipedia + BooksCorpus."
         ),
         "weights_url": "https://storage.googleapis.com/keras-nlp/models/bert_medium_uncased_en/model.h5",
         "weights_hash": "bb990e1184ec6b6185450c73833cd661",
+        "vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_medium_uncased_en/vocab.txt",
+        "vocabulary_hash": "64800d5d8528ce344256daf115d4965e",
     },
     "bert_base_uncased_en": {
         "config": {
@@ -83,13 +95,17 @@ backbone_presets = {
             "max_sequence_length": 512,
             "num_segments": 2,
         },
-        "vocabulary": "uncased_en",
+        "preprocessor_config": {
+            "lowercase": True,
+        },
         "description": (
             "Base size of BERT where all input is lowercased. "
             "Trained on English Wikipedia + BooksCorpus."
         ),
         "weights_url": "https://storage.googleapis.com/keras-nlp/models/bert_base_uncased_en/model.h5",
         "weights_hash": "9b2b2139f221988759ac9cdd17050b31",
+        "vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_base_uncased_en/vocab.txt",
+        "vocabulary_hash": "64800d5d8528ce344256daf115d4965e",
     },
     "bert_base_cased_en": {
         "config": {
@@ -102,13 +118,17 @@ backbone_presets = {
             "max_sequence_length": 512,
             "num_segments": 2,
         },
-        "vocabulary": "cased_en",
+        "preprocessor_config": {
+            "lowercase": False,
+        },
         "description": (
             "Base size of BERT where case is maintained. "
             "Trained on English Wikipedia + BooksCorpus."
         ),
         "weights_url": "https://storage.googleapis.com/keras-nlp/models/bert_base_cased_en/model.h5",
         "weights_hash": "f94a6cb012e18f4fb8ec92abb91864e9",
+        "vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_base_cased_en/vocab.txt",
+        "vocabulary_hash": "bb6ca9b42e790e5cd986bbb16444d0e0",
     },
     "bert_base_zh": {
         "config": {
@@ -121,10 +141,14 @@ backbone_presets = {
             "max_sequence_length": 512,
             "num_segments": 2,
         },
-        "vocabulary": "zh",
+        "preprocessor_config": {
+            "lowercase": False,
+        },
         "description": ("Base size of BERT. Trained on Chinese Wikipedia."),
         "weights_url": "https://storage.googleapis.com/keras-nlp/models/bert_base_zh/model.h5",
         "weights_hash": "79afa421e386076e62ab42dad555ab0c",
+        "vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_base_zh/vocab.txt",
+        "vocabulary_hash": "3b5b76c4aef48ecf8cb3abaafe960f09",
     },
     "bert_base_multi_cased": {
         "config": {
@@ -137,10 +161,17 @@ backbone_presets = {
             "max_sequence_length": 512,
             "num_segments": 2,
         },
-        "vocabulary": "multi_cased",
-        "description": ("Base size of BERT. Trained on Chinese Wikipedia."),
+        "preprocessor_config": {
+            "lowercase": False,
+        },
+        "description": (
+            "Base size of BERT. Trained on trained on Wikipedias of 104 "
+            "languages."
+        ),
         "weights_url": "https://storage.googleapis.com/keras-nlp/models/bert_base_multi_cased/model.h5",
         "weights_hash": "b0631cec0a1f2513c6cfd75ba29c33aa",
+        "vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_base_multi_cased/vocab.txt",
+        "vocabulary_hash": "d9d865138d17f1958502ed060ecfeeb6",
     },
     "bert_large_uncased_en": {
         "config": {
@@ -153,13 +184,17 @@ backbone_presets = {
             "max_sequence_length": 512,
             "num_segments": 2,
         },
-        "vocabulary": "uncased_en",
+        "preprocessor_config": {
+            "lowercase": True,
+        },
         "description": (
             "Large size of BERT where all input is lowercased. "
             "Trained on English Wikipedia + BooksCorpus."
         ),
         "weights_url": "https://storage.googleapis.com/keras-nlp/models/bert_large_uncased_en/model.h5",
         "weights_hash": "cc5cacc9565ef400ee4376105f40ddae",
+        "vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_large_uncased_en/vocab.txt",
+        "vocabulary_hash": "64800d5d8528ce344256daf115d4965e",
     },
     "bert_large_cased_en": {
         "config": {
@@ -172,60 +207,16 @@ backbone_presets = {
             "max_sequence_length": 512,
             "num_segments": 2,
         },
-        "vocabulary": "cased_en",
+        "preprocessor_config": {
+            "lowercase": False,
+        },
         "description": (
             "Base size of BERT where case is maintained. "
             "Trained on English Wikipedia + BooksCorpus."
         ),
         "weights_url": "https://storage.googleapis.com/keras-nlp/models/bert_large_cased_en/model.h5",
         "weights_hash": "8b8ab82290bbf4f8db87d4f100648890",
-    },
-}
-
-
-# Metadata for loading pretrained tokenizer vocabularies.
-# We need the vocabulary_size hardcoded so we can instantiate a BERT network
-# with the right embedding size without downloading the matching vocabulary.
-# TODO(mattdangerw): Update our bucket structure so the vocabularies are
-# stored in an independent way, rather than reading from the base model.
-vocabularies = {
-    "uncased_en": {
-        "description": (
-            "The vocabulary for BERT models trained on "
-            "English Wikipedia + BooksCorpus where case is discarded."
-        ),
-        "vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_base_uncased_en/vocab.txt",
-        "vocabulary_hash": "64800d5d8528ce344256daf115d4965e",
-        "vocabulary_size": 30522,
-        "lowercase": True,
-    },
-    "cased_en": {
-        "description": (
-            "The vocabulary for BERT models trained on "
-            "English Wikipedia + BooksCorpus where case is maintained."
-        ),
-        "vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_base_cased_en/vocab.txt",
+        "vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_large_cased_en/vocab.txt",
         "vocabulary_hash": "bb6ca9b42e790e5cd986bbb16444d0e0",
-        "vocabulary_size": 28996,
-        "lowercase": False,
-    },
-    "zh": {
-        "description": (
-            "The vocabulary for BERT models trained on Chinese Wikipedia."
-        ),
-        "vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_base_zh/vocab.txt",
-        "vocabulary_hash": "3b5b76c4aef48ecf8cb3abaafe960f09",
-        "vocabulary_size": 21128,
-        "lowercase": False,
-    },
-    "multi_cased": {
-        "description": (
-            "The vocabulary for BERT models trained on trained on Wikipedias "
-            "of 104 languages."
-        ),
-        "vocabulary_url": "https://storage.googleapis.com/keras-nlp/models/bert_base_multi_cased/vocab.txt",
-        "vocabulary_hash": "d9d865138d17f1958502ed060ecfeeb6",
-        "vocabulary_size": 119547,
-        "lowercase": False,
     },
 }
diff --git a/keras_nlp/models/utils.py b/keras_nlp/models/utils.py
new file mode 100644
index 00000000..423749c4
--- /dev/null
+++ b/keras_nlp/models/utils.py
@@ -0,0 +1,19 @@
+# Copyright 2022 The KerasNLP Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     https://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Utilities for models and preprocessors."""
+
+
+class classproperty(property):
+    def __get__(self, _, owner_cls):
+        return self.fget(owner_cls)


Test patch:
diff --git a/keras_nlp/models/bert/bert_preprocessing_test.py b/keras_nlp/models/bert/bert_preprocessing_test.py
index 5c6be8e0..134be921 100644
--- a/keras_nlp/models/bert/bert_preprocessing_test.py
+++ b/keras_nlp/models/bert/bert_preprocessing_test.py
@@ -13,6 +13,8 @@
 # limitations under the License.
 """Tests for BERT preprocessing layers."""
 
+import unittest
+
 import tensorflow as tf
 
 from keras_nlp.models.bert.bert_preprocessing import BertPreprocessor
@@ -46,11 +48,50 @@ class BertPreprocessorTest(tf.test.TestCase):
         self.assertAllEqual(output["token_ids"], [2, 9, 10, 11, 12, 1, 3, 0])
 
     def test_detokenize(self):
-        input_data = [[5, 6, 7, 8]]
+        input_tokens = [[5, 6, 7, 8]]
         preprocessor = BertPreprocessor(vocabulary=self.vocab)
-        output = preprocessor.tokenizer.detokenize(input_data)
+        output = preprocessor.tokenizer.detokenize(input_tokens)
         self.assertAllEqual(output, ["THE QUICK BROWN FOX"])
 
     def test_vocabulary_size(self):
         preprocessor = BertPreprocessor(vocabulary=self.vocab)
         self.assertEqual(preprocessor.vocabulary_size(), 13)
+
+    def test_unknown_preset_error(self):
+        # Not a preset name
+        with self.assertRaises(ValueError):
+            BertPreprocessor.from_preset("bert_base_uncased_clowntown")
+
+    @unittest.mock.patch("tensorflow.keras.utils.get_file")
+    def test_valid_call_presets(self, get_file_mock):
+        """Ensure presets have necessary structure, but no RCPs."""
+        input_data = ["THE QUICK BROWN FOX."]
+        get_file_mock.return_value = self.vocab
+        for preset in BertPreprocessor.presets:
+            preprocessor = BertPreprocessor.from_preset(preset)
+            preprocessor(input_data)
+        self.assertEqual(
+            get_file_mock.call_count, len(BertPreprocessor.presets)
+        )
+
+    @unittest.mock.patch("tensorflow.keras.utils.get_file")
+    def test_override_preprocessor_sequence_length(self, get_file_mock):
+        get_file_mock.return_value = self.vocab
+        preprocessor = BertPreprocessor.from_preset(
+            "bert_base_uncased_en",
+            sequence_length=64,
+        )
+        self.assertEqual(preprocessor.get_config()["sequence_length"], 64)
+        preprocessor("The quick brown fox.")
+        get_file_mock.assert_called_once()
+
+    @unittest.mock.patch("tensorflow.keras.utils.get_file")
+    def test_override_preprocessor_sequence_length_gt_max(self, get_file_mock):
+        """Override sequence length longer than model's maximum."""
+        get_file_mock.return_value = self.vocab
+        with self.assertRaises(ValueError):
+            BertPreprocessor.from_preset(
+                "bert_base_uncased_en",
+                sequence_length=1024,
+            )
+        get_file_mock.assert_called_once()
diff --git a/keras_nlp/tests/network_tests/test_load_presets.py b/keras_nlp/tests/network_tests/test_load_presets.py
index 2718fec3..cf8d9745 100644
--- a/keras_nlp/tests/network_tests/test_load_presets.py
+++ b/keras_nlp/tests/network_tests/test_load_presets.py
@@ -16,17 +16,16 @@
 import pytest
 import tensorflow as tf
 
-from keras_nlp.models.bert import bert_models
-from keras_nlp.models.bert import bert_preprocessing
-from keras_nlp.models.bert import bert_presets
-from keras_nlp.models.bert import bert_tasks
+from keras_nlp.models.bert.bert_models import Bert
+from keras_nlp.models.bert.bert_preprocessing import BertPreprocessor
+from keras_nlp.models.bert.bert_tasks import BertClassifier
 
 
 @pytest.mark.slow
 class BertPresetTest(tf.test.TestCase):
     def test_load_bert(self):
-        for preset in bert_models.Bert.presets:
-            model = bert_models.Bert.from_preset(preset, load_weights=True)
+        for preset in Bert.presets:
+            model = Bert.from_preset(preset, load_weights=True)
             input_data = {
                 "token_ids": tf.random.uniform(
                     shape=(1, 512), dtype=tf.int64, maxval=model.vocabulary_size
@@ -39,8 +38,8 @@ class BertPresetTest(tf.test.TestCase):
             model(input_data)
 
     def test_load_bert_backbone_string(self):
-        for preset in bert_models.Bert.presets:
-            classifier = bert_tasks.BertClassifier(preset, 4, name="classifier")
+        for preset in Bert.presets:
+            classifier = BertClassifier(preset, 4, name="classifier")
             input_data = {
                 "token_ids": tf.random.uniform(
                     shape=(1, 512),
@@ -55,7 +54,7 @@ class BertPresetTest(tf.test.TestCase):
             classifier(input_data)
 
     def test_classifier_default_args(self):
-        classifier = bert_tasks.BertClassifier()
+        classifier = BertClassifier()
         input_data = {
             "token_ids": tf.random.uniform(
                 shape=(1, 512),
@@ -67,9 +66,7 @@ class BertPresetTest(tf.test.TestCase):
         }
         classifier(input_data)
 
-    def test_load_vocabularies(self):
-        for vocabulary in bert_presets.vocabularies:
-            tokenizer = bert_preprocessing.BertPreprocessor(
-                vocabulary=vocabulary,
-            )
-            tokenizer("The quick brown fox.")
+    def test_load_preprocessors(self):
+        for preset in BertPreprocessor.presets:
+            preprocessor = BertPreprocessor.from_preset(preset)
+            preprocessor("The quick brown fox.")