Package: cclm
CCLMComposable, Character-Level ModelsWhycclm?The goal ofcclmis to make the deep learning model development process modular by providing abstractions for structuring a computational graph.If we think of the ML lifecycle as producing a usable classModelthat consumers can call oninputto getoutput, then comparing the model training process to human-led software development highlights some big differences. For instance often when we retrain models, we usually change the whole model at once - imagine a developer telling you every commit they made touched every line of code in the package. Similarly, using a pretrained model is like using a 'batteries included' framework: you likely end up inheriting a good deal of functionality you don't require, and it may be hard to customize. These differences suggest that there may be changes that could make it easier to manage deep learning model development, particularly as models continue to explode in size.How does it work?The waycclmaims to achieve the above is by making the model building process composable. There are many ways to pretrain a model on text, and infinite corpora on which to train, and each application has different needs.cclmmakes it possible to define abaseinput on which to build many different computational graphs, then combine them. For instance, if there is a standard, publishedcclmmodel trained with masked language modeling (MLM) on (wikitext+bookcorpus), you might start with that, but add a second component to that model that uses the samebase, but is pretrained to extract entities fromwiki-ner. By combining the two pretrained components with aComposedModel, you get a model with information from both tasks that you can then use as a starting point for your downstream task.Common model components will be published onto thecclm-shelfto make it simple to mix and match capabilities.The choice to emphasize character-level rather than arbitrary tokenization schemes is to make the input as generically useful across tasks as possible. Character-level input also makes it simpler to add realistic typos/noise to make models more robust to imperfect inputs.Basic conceptsThe main output of a training job withcclmis aComposedModel, which consists of aPreprocessorthat turns text into a vector[int], a base model that embeds that vector input, and one or more model components that accept the output of the embedder. TheComposedModelconcatenates the output from those models together to produce its final output.The package usesdatasetsandtokenizersfromhuggingfacefor a standard interface and to benefit from their great framework. To fit models and preprocessors, you can also pass aList[str]directly.To start, you need aPreprocessor.fromcclm.preprocessingimportPreprocessorprep=Preprocessor()# set max_example_len to specify a maximum input lengthprep.fit(dataset)# defines the model's vocabulary (character-level)Once you have that, you can create anEmbedder, which is the common base on which all the separate models will sit. This is a flexible class primarily responsible for holding a model that embeds a sequence of integers (representing characters) into a space the components expect. For more complicated setups, theEmbeddercould have aComposedModelas itsmodelfromcclm.modelsimportEmbedderembedder=Embedder(prep.max_example_len,prep.n_chars)The embedder doesn't necessarily need to be fit by itself, as you can fit it while you do your first pretraining task.Now you're ready to build your first model using a pretraining task (here masked language modeling)fromcclm.pretrainingimportMaskedLanguagePretrainerpretrainer=MaskedLanguagePretrainer(embedder=embedder)pretrainer.fit(dataset,epochs=10)TheMaskedLanguagePretrainerdefines a transformer-based model to do masked language modeling. Calling.fit()will use thePreprocessorto produce masked inputs and try to identify the missing input token(s) usingsampled_softmaxloss or negative sampling. This is just one example of a pretraining task, but others can be found incclm.pretrainers.Once you've trained one or more models usingPretrainerobjects, you can compose them together into one model.composed=ComposedModel(embedder,[pretrainer_a.model,pretrainer_b.model])You can then usecomposed.model(x)to embed inputx=prep.string_to_array("cclm is neat",prep.max_example_len)emb=composed.model(x)# has shape (1, prep.max_example_len, pretrainer_a_model_shape[-1]+pretrainer_b_model_shape[-1])... or create a new model with something like# pool the output across the character dimensiongmp=tf.keras.layers.GlobalMaxPool1D()# add a classification head on topd=tf.keras.layers.Dense(1,activation="sigmoid")keras_model=tf.keras.Model(composed.model.input,d(gmp(composed.model.output)))ShelfTheShelfclass is used to load off-the-shelf components. These are published to a separate repo using git lfs, and are loaded with a specific tag.fromcclm.shelfimportShelfshelf=Shelf()identifier="en_wiki_clm_1"item_type="preprocessor"shelf.fetch(identifier,item_type,tag="v0.2.1",cache_dir=".cclm")prep=Preprocessor(load_from=os.path.join(cache_dir,identifier,item_type,"cclm_config.json"))