Package: sklearn-instrumentation
Generalized instrumentation tooling for scikit-learn models.sklearn_instrumentationallows instrumenting thesklearnpackage and any scikit-learn compatible packages with estimators and transformers inheriting fromsklearn.base.BaseEstimator.Instrumentation applies decorators to methods ofBaseEstimator-derived classes or instances. By default the instrumentor applies instrumentation to the following methods (except when they are properties of instances):fitfit_transformpredictpredict_log_probapredict_probatransform_fit_fit_transform_predict_predict_log_proba_predict_proba_transformsklearn-instrumentationsupports instrumentation of full sklearn-compatible packages, as well as recursive instrumentation of models (metaestimators likePipeline, or even single estimators likeRandomForestClassifier)InstallationThe sklearn-instrumentation package is available on pypi and can be installed using pippipinstallsklearn-instrumentationPackage instrumentationInstrument any sklearn compatible package that hasBaseEstimator-derived classes.fromsklearn_instrumentationimportSklearnInstrumentorinstrumentor=SklearnInstrumentor(instrument=my_instrument)instrumentor.instrument_packages(["sklearn","xgboost","lightgbm"])Full example:importloggingfromsklearn.datasetsimportload_irisfromsklearn.decompositionimportPCAfromsklearn.ensembleimportRandomForestClassifierfromsklearn.pipelineimportFeatureUnionfromsklearn.pipelineimportPipelinefromsklearn.preprocessingimportStandardScalerfromsklearn_instrumentationimportSklearnInstrumentorfromsklearn_instrumentation.instruments.loggingimportTimeElapsedLoggerlogging.basicConfig(level=logging.INFO)# Create an instrumentor and instrument sklearninstrumentor=SklearnInstrumentor(instrument=TimeElapsedLogger())instrumentor.instrument_packages(["sklearn"])# Create a toy model for classificationss=StandardScaler()pca=PCA(n_components=3)rf=RandomForestClassifier()classification_model=Pipeline(steps=[("fu",FeatureUnion(transformer_list=[("ss",ss),("pca",pca),]),),("rf",rf),])X,y=load_iris(return_X_y=True)# Observe loggingclassification_model.fit(X,y)# INFO:sklearn_instrumentation.instruments.logging:Pipeline.fit starting.# INFO:sklearn_instrumentation.instruments.logging:Pipeline._fit starting.# INFO:sklearn_instrumentation.instruments.logging:StandardScaler.fit starting.# INFO:sklearn_instrumentation.instruments.logging:StandardScaler.fit elapsed time: 0.0006406307220458984 seconds# INFO:sklearn_instrumentation.instruments.logging:StandardScaler.transform starting.# INFO:sklearn_instrumentation.instruments.logging:StandardScaler.transform elapsed time: 0.0001430511474609375 seconds# INFO:sklearn_instrumentation.instruments.logging:PCA._fit starting.# INFO:sklearn_instrumentation.instruments.logging:PCA._fit elapsed time: 0.0006711483001708984 seconds# INFO:sklearn_instrumentation.instruments.logging:Pipeline._fit elapsed time: 0.0026731491088867188 seconds# INFO:sklearn_instrumentation.instruments.logging:BaseForest.fit starting.# INFO:sklearn_instrumentation.instruments.logging:BaseForest.fit elapsed time: 0.1768970489501953 seconds# INFO:sklearn_instrumentation.instruments.logging:Pipeline.fit elapsed time: 0.17983102798461914 seconds# Observe loggingclassification_model.predict(X)# INFO:sklearn_instrumentation.instruments.logging:Pipeline.predict starting.# INFO:sklearn_instrumentation.instruments.logging:FeatureUnion.transform starting.# INFO:sklearn_instrumentation.instruments.logging:StandardScaler.transform starting.# INFO:sklearn_instrumentation.instruments.logging:StandardScaler.transform elapsed time: 0.00024509429931640625 seconds# INFO:sklearn_instrumentation.instruments.logging:_BasePCA.transform starting.# INFO:sklearn_instrumentation.instruments.logging:_BasePCA.transform elapsed time: 0.0002181529998779297 seconds# INFO:sklearn_instrumentation.instruments.logging:FeatureUnion.transform elapsed time: 0.0012080669403076172 seconds# INFO:sklearn_instrumentation.instruments.logging:ForestClassifier.predict starting.# INFO:sklearn_instrumentation.instruments.logging:ForestClassifier.predict_proba starting.# INFO:sklearn_instrumentation.instruments.logging:ForestClassifier.predict_proba elapsed time: 0.013531208038330078 seconds# INFO:sklearn_instrumentation.instruments.logging:ForestClassifier.predict elapsed time: 0.013692140579223633 seconds# INFO:sklearn_instrumentation.instruments.logging:Pipeline.predict elapsed time: 0.015219926834106445 seconds# Remove instrumentationinstrumentor.uninstrument_packages(["sklearn"])# Observe no loggingclassification_model.predict(X)Instance instrumentationInstrument any sklearn compatible trained estimator or metaestimator.fromsklearn_instrumentationimportSklearnInstrumentorinstrumentor=SklearnInstrumentor(instrument=my_instrument)instrumentor.instrument_instance(estimator=my_ml_pipeline)Example:importloggingfromsklearn.datasetsimportload_irisfromsklearn_instrumentationimportSklearnInstrumentorfromsklearn_instrumentation.instruments.loggingimportTimeElapsedLoggerfromsklearn.ensembleimportRandomForestClassifierlogging.basicConfig(level=logging.INFO)# Train a classifierX,y=load_iris(return_X_y=True)rf=RandomForestClassifier()rf.fit(X,y)# Create an instrumentor which decorates BaseEstimator methods with# logging output when entering and exiting methods, with time elapsed logged# on exit.instrumentor=SklearnInstrumentor(instrument=TimeElapsedLogger())# Apply the decorator to all BaseEstimators in each of these librariesinstrumentor.instrument_instance(rf)# Observe the logging outputrf.predict(X)# INFO:sklearn_instrumentation.instruments.logging:ForestClassifier.predict starting.# INFO:sklearn_instrumentation.instruments.logging:ForestClassifier.predict_proba starting.# INFO:sklearn_instrumentation.instruments.logging:ForestClassifier.predict_proba elapsed time: 0.014165163040161133 seconds# INFO:sklearn_instrumentation.instruments.logging:ForestClassifier.predict elapsed time: 0.014327764511108398 seconds# Remove the decorator from all BaseEstimators in each of these librariesinstrumentor.uninstrument_instance(rf)# No more loggingrf.predict(X)Instance class instrumentationDuring fitting, some metaestimators will copy estimator instances using scikit-learn’sclonefunction. This results in cloned fitted estimators not having instrumentation. To get around this we can instrument the classes rather than the instances.fromsklearn.datasetsimportload_irisfromsklearn.decompositionimportPCAfromsklearn.ensembleimportRandomForestClassifierfromsklearn.pipelineimportFeatureUnionfromsklearn.pipelineimportPipelinefromsklearn.preprocessingimportStandardScalerfromsklearn_instrumentationimportSklearnInstrumentorfromsklearn_instrumentation.instruments.loggingimportTimeElapsedLoggerlogging.basicConfig(level=logging.INFO)ss=StandardScaler()pca=PCA(n_components=3)rf=RandomForestClassifier()classification_model=Pipeline(steps=[("fu",FeatureUnion(transformer_list=[("ss",ss),("pca",pca),]),),("rf",rf),])X,y=load_iris(return_X_y=True)instrumentor=SklearnInstrumentor(instrument=TimeElapsedLogger())instrumentor.instrument_instance_classes(classification_model)classification_model.fit(X,y)# INFO:sklearn_instrumentation.instruments.logging:Pipeline.fit starting.# INFO:sklearn_instrumentation.instruments.logging:Pipeline.fit starting.# INFO:sklearn_instrumentation.instruments.logging:Pipeline._fit starting.# INFO:sklearn_instrumentation.instruments.logging:Pipeline._fit starting.# INFO:sklearn_instrumentation.instruments.logging:StandardScaler.fit starting.# INFO:sklearn_instrumentation.instruments.logging:StandardScaler.fit starting.# INFO:sklearn_instrumentation.instruments.logging:StandardScaler.fit elapsed time: 0.0006749629974365234 seconds# INFO:sklearn_instrumentation.instruments.logging:StandardScaler.fit elapsed time: 0.0007731914520263672 seconds# INFO:sklearn_instrumentation.instruments.logging:StandardScaler.transform starting.# INFO:sklearn_instrumentation.instruments.logging:StandardScaler.transform starting.# INFO:sklearn_instrumentation.instruments.logging:StandardScaler.transform elapsed time: 0.00016427040100097656 seconds# INFO:sklearn_instrumentation.instruments.logging:StandardScaler.transform elapsed time: 0.0002810955047607422 seconds# INFO:sklearn_instrumentation.instruments.logging:PCA._fit starting.# INFO:sklearn_instrumentation.instruments.logging:PCA._fit starting.# INFO:sklearn_instrumentation.instruments.logging:PCA._fit elapsed time: 0.0004239082336425781 seconds# INFO:sklearn_instrumentation.instruments.logging:PCA._fit elapsed time: 0.0005612373352050781 seconds# INFO:sklearn_instrumentation.instruments.logging:Pipeline._fit elapsed time: 0.002705097198486328 seconds# INFO:sklearn_instrumentation.instruments.logging:Pipeline._fit elapsed time: 0.002802133560180664 seconds# INFO:sklearn_instrumentation.instruments.logging:BaseForest.fit starting.# INFO:sklearn_instrumentation.instruments.logging:BaseForest.fit starting.# INFO:sklearn_instrumentation.instruments.logging:BaseForest.fit elapsed time: 0.16085195541381836 seconds# INFO:sklearn_instrumentation.instruments.logging:BaseForest.fit elapsed time: 0.16097569465637207 seconds# INFO:sklearn_instrumentation.instruments.logging:Pipeline.fit elapsed time: 0.1639721393585205 seconds# INFO:sklearn_instrumentation.instruments.logging:Pipeline.fit elapsed time: 0.16404390335083008 secondsclassification_model.predict(X)# INFO:sklearn_instrumentation.instruments.logging:Pipeline.predict starting.# INFO:sklearn_instrumentation.instruments.logging:Pipeline.predict starting.# INFO:sklearn_instrumentation.instruments.logging:FeatureUnion.transform starting.# INFO:sklearn_instrumentation.instruments.logging:FeatureUnion.transform starting.# INFO:sklearn_instrumentation.instruments.logging:StandardScaler.transform starting.# INFO:sklearn_instrumentation.instruments.logging:StandardScaler.transform starting.# INFO:sklearn_instrumentation.instruments.logging:StandardScaler.transform elapsed time: 0.0001049041748046875 seconds# INFO:sklearn_instrumentation.instruments.logging:StandardScaler.transform elapsed time: 0.00017309188842773438 seconds# INFO:sklearn_instrumentation.instruments.logging:_BasePCA.transform starting.# INFO:sklearn_instrumentation.instruments.logging:_BasePCA.transform starting.# INFO:sklearn_instrumentation.instruments.logging:_BasePCA.transform elapsed time: 0.0001690387725830078 seconds# INFO:sklearn_instrumentation.instruments.logging:_BasePCA.transform elapsed time: 0.00023698806762695312 seconds# INFO:sklearn_instrumentation.instruments.logging:FeatureUnion.transform elapsed time: 0.0008630752563476562 seconds# INFO:sklearn_instrumentation.instruments.logging:FeatureUnion.transform elapsed time: 0.0009222030639648438 seconds# INFO:sklearn_instrumentation.instruments.logging:ForestClassifier.predict starting.# INFO:sklearn_instrumentation.instruments.logging:ForestClassifier.predict starting.# INFO:sklearn_instrumentation.instruments.logging:ForestClassifier.predict_proba starting.# INFO:sklearn_instrumentation.instruments.logging:ForestClassifier.predict_proba starting.# INFO:sklearn_instrumentation.instruments.logging:ForestClassifier.predict_proba elapsed time: 0.01138925552368164 seconds# INFO:sklearn_instrumentation.instruments.logging:ForestClassifier.predict_proba elapsed time: 0.011497974395751953 seconds# INFO:sklearn_instrumentation.instruments.logging:ForestClassifier.predict elapsed time: 0.011577844619750977 seconds# INFO:sklearn_instrumentation.instruments.logging:ForestClassifier.predict elapsed time: 0.011635780334472656 seconds# INFO:sklearn_instrumentation.instruments.logging:Pipeline.predict elapsed time: 0.012682199478149414 seconds# INFO:sklearn_instrumentation.instruments.logging:Pipeline.predict elapsed time: 0.012733936309814453 secondsinstrumentor.uninstrument_instance_classes(classification_model)classification_model.predict(X)InstrumentsThe package comes with a handful of instruments which log information aboutXor timing of execution. You can create your own instrument just by creating a decorator, following this patternfromfunctoolsimportwrapsdefmy_instrumentation(estimator,func,**dkwargs):"""Wrap an estimator method with instrumentation.

    :param obj: The class or instance on which to apply instrumentation
    :param func: The method to be instrumented.
    :param dkwargs: Decorator kwargs, which can be passed to the
        decorator at decoration time. For estimator instrumentation
        this allows different parametrizations for each ml model.
    """@wraps(func)defwrapper(*args,**kwargs):"""Wrapping function.

        :param args: The args passed to methods, typically
            just ``X`` and/or ``y``
        :param kwargs: The kwargs passed to methods, usually
            weights or other params
        """# Code goes here before execution of the estimator methodretval=func(*args,**kwargs)# Code goes here after execution of the estimator methodreturnretvalreturnwrapperTo create a stateful instrument, use a class with the__call__method for implementing the decorator:fromfunctoolsimportwrapsfromsklearn_instrumentation.instruments.baseimportBaseInstrumentclassMyInstrument(BaseInstrument)def__init__(self,*args,**kwargs):# handle any statefulness herepassdef__call__(self,estimator,func,**dkwargs):"""Wrap an estimator method with instrumentation.

        :param obj: The class or instance on which to apply instrumentation
        :param func: The method to be instrumented.
        :param dkwargs: Decorator kwargs, which can be passed to the
            decorator at decoration time. For estimator instrumentation
            this allows different parametrizations for each ml model.
        """@wraps(func)defwrapper(*args,**kwargs):"""Wrapping function.

            :param args: The args passed to methods, typically
                just ``X`` and/or ``y``
            :param kwargs: The kwargs passed to methods, usually
                weights or other params
            """# Code goes here before execution of the estimator methodretval=func(*args,**kwargs)# Code goes here after execution of the estimator methodreturnretvalreturnwrapperTo pass kwargs for different ml models:instrumentor=SklearnInstrumentor(instrument=my_instrument)instrumentor.instrument_instance(estimator=ml_model_1,instrument_kwargs={"name":"awesome_model"})instrumentor.instrument_instance(estimator=ml_model_2,instrument_kwargs={"name":"better_model"})