Package: kedro-boot
What is kedro-boot ?Kedro Boot makes it simple to integrate your Kedro pipelines with any applications. It's a framework for creating APIs and SDKs for you kedro projects.
It offers these key functionalities:Data injection: Streamline the process of feeding data from any application into Kedro pipelinesLow-Latency Execution: Execute multiple pipeline runs with minimal delay, optimising for time-sensitive web applications.This enable using Kedro pipelines in a wide range of use cases, including model serving, data apps (streamlit, dash), statistical simulations, paralell processing of unstructured data, streaming, and more.  It also streamline the deployment of your Kedro pipeline into various Data & AI PlatformsIf you're new tokedro, we invite you to visitkedro docsHow do I use Kedro Boot ?Any application can consume kedro pipelines through REST APIs or as a library (SDK). Kedro Boot provides utilities and abstractions for each of these integration patternsConsuming Kedro pipeline through REST APIYou can serve your kedro pipelines as a REST API using Kedro FastAPI ServerFirst you should install kedro-boot with fastapi extra dependencypip install kedro-boot[fastapi]Then you can serve your fastapi app with :kedro boot fastapi --app path.to.your.fastapi.app <kedro_run_args>Your fastapi app objects will be mapped with kedro pipeline objects, and the run results will be injected into your KedroFastAPI object throughFastAPI dependency injection. Here is an illustration of the kedro <-> fastapi objects mapping:A default FastAPI app is used if no FastAPI app given. It would serve a single endpoint that run in background your selected pipelinekedro boot fastapi <kedro_run_args>These production-ready features would be natively included in your FastAPI apps:EmbededGunicron web server(only for linux and macos)Pyctuatorthat report some service health metrology and application states. Usually used by service orchestrators (kubernetes) or monitoring to track service health and ensure it's high availabilityMultiple environments configurations, leveraging kedro's OmegaConfigLoader.["fastapi*/"]config pattern could be used to configure the web server. Configs could also be passed as CLI args (refer to --help)You can learn more by testing thespaceflights Kedro FastAPI examplethat showcases serving multiples endpoints operations that are mapped to differents pipeline namespacesConsuming Kedro pipelines through SDKAny python applications could consume kedro pipeline as a library. The integration process involves two steps:Registring kedro pipelineCreating a Kedro Boot SessionRegistring Kedro pipelinesKedro Boot prepare the catalog for the application consumption through a compilation process that follow a compilation specs.Compilation specs defines the namespaces and their datasets (inputs, outputs, parameters) that would be exposed to the application. It specify also if artifacts datasets should be infered during the compilation process (artifacts datasets are loaded in MemoryDataset at runtime in order to speed up iteration time)The compilation specs are either given by the Application or infered from the Kedro Pipeline.Here is an example of registring a pipeline that contains inference and evaluation namespaces.# create inference namespace. All the inference pipeline's datasets will be exposed to the app, except "regressor" and "model_options.inference_pipeline=pipeline([features_nodes,prediction_nodes],inputs={"regressor":"training.regressor"},parameters="model_options",namespace="inference",)# create evaluation namespace. All the evaluation pipeline's datasets will be exposed to the app, except "feature_store", "regressor" and "model_options.evaluation_pipeline=pipeline([model_input_nodes,prediction_nodes,evaluation_nodes],inputs={"features_store":"features_store","regressor":"training.regressor"},parameters="model_options",namespace="evaluation",)spaceflights_pipelines=inference_pipeline+evaluation_pipelinereturn{"__default__":spaceflights_pipelines}In this example, all the namespaces and their namespaced datasets (inputs, outputs, parameters) would infer compilation specs and therefore would be exposed to the Application.You can use kedro-viz to visualize the datasets that woulc be exposed to the kedro boot apps. In the figure below, for theinferencenamespace, we see clearly thatinference.feature_storeandinference.predictionswill be exposed to the applicaton (the blue one).Below are the differents categories of datasets that forms the compiled catalog.Inputs: inputs datasets that are be injected by the app at iteration time.Outputs: outputs dataset that hold the run results.Parameters: parameters that are injected by the app at iteration time.Artifacts: artifacts datasets that are materialized (loaded as MemoryDataset) at startup time.Templates: template datasets that contains ${itertime_params: param_name}. Their attributes are interpolated at iteration time.You can compile the catalog without actually using it in a Kedro Boot App. This is helpful for verifying if the expected artifacts datasets are correctly infered or if the template datasets are correctly detected. Here is an example of the catalog compilation report for a pipeline that contains aninferencenamespace.kedro boot compileCompilation results:INFO  catalog compilation completed for the namespace 'inference'. Here is the report:                                                          
    - Input datasets to be replaced/rendered at iteration time: {'inference.features_store'}
    - Output datasets that hold the results of a run at iteration time: {'inference.predictions'}
    - Parameter datasets to be replaced/rendered at iteration time: set()
    - Artifact datasets to be materialized (preloader as memory dataset) at startup time: {'training.regressor'}
    - Template datasets to be rendered at iteration time: set()
		
INFO  Catalog compilation completed.We can see that thetraining.regressoris beign infered as artifact, it will be loaded as memory dataset to speed up iterations and prevent memory leak in a web app use case.Note that when infering compilation specs, a pipeline that have no namespaces is also exposed to the kedro boot apps (have a compilation spec), but does not expose any datasets. Applications could provide their own compilation specs in order to specify the datasets that are needed to be exposed.Creating Kedro Boot SessionThe Kedro Boot Session could be created from within an external application usingboot_projectorboot_packagefactories, or it can be created inside the kedro project using theAbstractKedroBootAppin the case of an embeded application.Standalone mode: The application hold the entry pointThis refer to applications that have their own CLI entry points and cannot be embeded in kedro's entry point. Streamlit and Data & AI Platforms are an example of such applications.These application could create a Kedro Boot Session from kedro project or kedro package, usingboot_projectorboot_packagefromkedro_boot.app.booterimportboot_projectfromkedro_boot.app.booterimportboot_packagefromkedro_boot.framework.compilation.specsimportCompilationSpecsession=boot_project(project_path="<your_project_path>",compilation_specs=[CompilationSpec(inputs=["your_dataset_name"])],# Would be infered if not givenkedro_args={# kedro run args"pipeline":"your_pipeline","conf_source":"<your_conf_source>",},)# session = boot_package(#     package_name="<your_package_name>",#     compilation_specs=[CompilationSpec(inputs=["your_dataset_name"])],#     kedro_args={#         "pipeline": "your_pipeline",#         "conf_source": "<your_conf_source>",#     },# )run_results=session.run(inputs={"your_dataset_name":your_data})You can found a complete example of a steamlit app that serve an ML model in theKedro Boot Examplesproject. We invite you to test it to gain a better understanding of Kedro Boot'sboot_projectorboot_packageinterfacesEmbedded mode : The application is embeded inside kedro projectThis mode involves using Kedro Boot to embed an application inside a Kedro project, leveraging kedro's entry points, session and config loader for managing application lifecycle. It's suitable for use cases when the application is lightweight and owned by the same team that developed the kedro pipelines.An embedded Kedro Boot App is an implementation of theAbstractKedroBootApp. The kedro Boot Session is provided through the_runabstractmethod.fromkedro_boot.appimportAbstractKedroBootAppfromkedro_boot.framework.sessionimportKedroBootSessionclassKedroBootApp(AbstractKedroBootApp):def_run(self,kedro_boot_session:KedroBootSession):# leveraging config_loader to manage app's configsmy_app_configs=kedro_boot_session.config_loader["my_app"]# You should delcare this config pattern in settings.pyfor_inmy_app_configs.get("num_iteration"):# Doing mutliples pipeline runskedro_boot_session.run(namespace="my_namespace",)The Kedro Boot App could be declared either in kedro's settings or as Kedro Boot run CLI argsDeclaring Kedro Boot App through settings.pyfromyour_package.your_moduleimportKedroBootAppAPP_CLASS=KedroBootAppAPP_ARGS={}# Any class init argsThe Kedro Boot App could be started with:kedro boot run <kedro_run_args>Declaring Kedro Boot App through kedro boot run CLI (Take precedence)kedro boot run --app path.to.your.KedroBootApp <kedro_run_args>You can find an example of aMonte Carlo App embeded into a kedro projectWhy does Kedro Boot exist ?Kedro Boot unlock the value of your kedro pipelines by giving you a structured way for integrating them in a larger application. We developed kedro-boot to achieve the following :Streamline deployment of kedro pipelines in batch, streaming and web app context.Encourage reuse and prevent rewriting pipeline's logic by the team that own the front applicationLeverage kedro's capabilities for business logic separation and authoringLeverage kedro's capabilities for managing an application lifecycleKedro Boot apps utilize Kedro's pipeline as a means to construct and manage business logic and some part or I/O. Kedro's underlying principles and internals ensure the maintainability, clarity, reuse, and visibility (kedro-viz) of business logic within Kedro Boot Apps, thanks to Kedro's declarative nature.Where do I test Kedro Boot ?You can refer to theKedro Boot Examplesproject; it will guide you through four examples of Kedro Boot usages.Can I contribute ?We'd be happy to receive help to maintain and improve the package. Any PR will be considered.