Package: kotsu
kotsu: lightweight framework for structuring model validationWhat is it?kotsuis Python package that provides a lightweight and flexible framework to structure
validating and comparing machine learning models. It aims to provide the skeleton on which to
develop models and to validate them in a robust and repeatable way,minimizing
bloat or overhead. Its flexibility allows usage withany model interfaceand any
validation technique, no matter how complex. The structure it providesavoids
common pitfallsthat occur when attempting to make fair comparisons between models.Main FeaturesRegister a model with hyperparameters to a unique IDRegister validations to a unique IDRun all registered models through all registered validations, and have the results compiled and
stored as a CSVOptionally passes anartefacts_store_dirto your validations, for storing of outputs
for further analysis, e.g. trained models or model predictions on test data setsDoesn't enforce any constraints or requirements on your models' interfacesPure Python package, with no other setup or configuration of other systems requiredWhere to get itThe source code is currently hosted on GitHub at:https://github.com/datavaluepeople/kotsuThe latest released version of the package can be installed from PyPI with:pipinstallkotsuUsageThe following demonstrates a simple usage of kotsu to register and validate multiple models over
multiple validations.Import kotsu and your packages for modelling:importkotsufromsklearnimportdatasets,svmfromsklearn.model_selectionimportcross_val_scoreRegister your competing models:Here we register two Support Vector Classifiers with different hyper-parameters.model_registry=kotsu.registration.ModelRegistry()model_registry.register(id="SVC-v1",entry_point=svm.SVC,kwargs={"kernel":"linear","C":1,"random_state":1},)model_registry.register(id="SVC-v2",entry_point=svm.SVC,kwargs={"kernel":"linear","C":0.5,"random_state":1},)Register your validations:You can register multiple validations if you want to compare models in different scenarios, e.g. on
different datasets. Your validations should take an instance of your models as an argument, then
return a dictionary containing the results from validation of that model. Here we register two
Cross-Validation validations with different numbers of folds.validation_registry=kotsu.registration.ValidationRegistry()deffactory_iris_cross_validation(folds:int):"""Factory for iris cross validation."""defiris_cross_validation(model)->dict:"""Iris classification cross validation."""X,y=datasets.load_iris(return_X_y=True)scores=cross_val_score(model,X,y,cv=folds)results={f"fold_{i}_score":scorefori,scoreinenumerate(scores)}results["mean_score"]=scores.mean()results["std_score"]=scores.std()returnresultsreturniris_cross_validationvalidation_registry.register(id="iris_cross_validation-v1",entry_point=factory_iris_cross_validation,kwargs={"folds":5},)validation_registry.register(id="iris_cross_validation-v2",entry_point=factory_iris_cross_validation,kwargs={"folds":10},)Run the models through the validations:We choose the current directory as the location for writing the results.kotsu.run(model_registry,validation_registry)Then find the results from each model-validation combination in a CSV written to the current
directory.Documentation on interfacesSeekotsu.typingfor
documentation on the main entities; Models, Validations, and Results, and their interfaces.Comprehensive exampleSee theend to end testfor a more comprehensive example usage of kotsu, which includes storing the trained models from
each model-validation run.LicenseMIT