Saravanan666/pypi-org huggingface dataset:
Package: kedro-cache
Kedro Cache:warning:This plugin is still under active developement and not fully tested. Do not use this in any production systems. Please report any issues that you find.📝 Descriptionkedro-cacheis akedroplugin that plugin that enables the caching of data sets.
The advantage is that the data sets are loaded from data catalog and not recomputed if they have not changed.
If the input data sets or code have changed, the outputs are recomputed and the data catalog is updated.
This plugin works out of the box with any kedro project without having to change the code.
The logic on how to determine if the cached data set in the catalog should be used is described in the flow chart below.Disclaimer:The caching strategy determines if a node function has changes by simply looking at the immediate function body.
This does not take into account other things such as called function, global variable etc. that might also have changed.🏆 FeaturesCaching of node outputs in catalogNo change to kedro project neededIntegration with kedro data catalogConfiguration viaconfig.ymlfile🏗 InstallationThe plugin can be install withpippipinstallkedro-cache🚀 Enable CachingIn the root directory of your kedro project, runkedrocacheinitThis will create a new filecache.ymlin theconfdirectory of your kedro project in which you can configure thekedro-cachemodule.
Although this step is optional as the plugin comes with default configurations.Next let's assume that you have the following kedro pipeline for which you want to add caching.
There are two nodes.
One that reads data from ainputdataset, does some computations and writes it to aintermediatedataset and one that reads the data from theintermediatedataset and writes it to theoutputdataset.# pipeline.pydefregister_pipelines()->Dict[str,Pipeline]:default_pipeline=pipeline([node(func=lambdax:x,inputs="input",outputs="intermediate",),node(func=lambdax:x,inputs="intermediate",outputs="output",),],)return{"__default__":default_pipeline}In order to add logging we simply just have to register all used data sets in the data catalog.
Because if the first node want to use the cached output instead of recalculating it, it need to load it from the data catalog.
This is only possible if it was stored there.# catalog.ymlinput:type:pandas.CSVDataSetfilepath:input.csvintermediate:type:pandas.CSVDataSetfilepath:intermediate.csvoutput:type:pandas.CSVDataSetfilepath:output.csvAnd that was it. Just by adding all files to the catalog you enabled caching.
Package: kedro-datasets
Kedro-DatasetsWelcome tokedro_datasets, the home of Kedro's data connectors. Here you will findAbstractDatasetimplementations powering Kedro's DataCatalog created by QuantumBlack and external contributors.Installationkedro-datasetsis a Python plugin. To install it:pipinstallkedro-datasetsWhatAbstractDatasetimplementations are supported?We support a range of data connectors, including CSV, Excel, Parquet, Feather, HDF5, JSON, Pickle, SQL Tables, SQL Queries, Spark DataFrames and more. We even allow support for working with images.These data connectors are supported with the APIs ofpandas,spark,networkx,matplotlib,yamland more.The Data Catalogallows you to work with a range of file formats on local file systems, network file systems, cloud object stores, and Hadoop.Here is a full list ofsupported data connectors and APIs.How can I create my ownAbstractDatasetimplementation?Take a look at ourinstructions on how to create your ownAbstractDatasetimplementation.Can I contribute?Yes! Want to help build Kedro-Datasets? Check out our guide tocontributing.What licence do you use?Kedro-Datasets is licensed under theApache 2.0License.Python version support policyTheKedro-Datasetspackage follows theNEP 29Python version support policy.
Package: kedro-diff
kedro-diffkedro-diffaims to be a familiar interface into comparing two points in
history.  Git diffs are fantastic tools but often are too granular to see what
has changed inside the pipeline.kedro-diffaims to be a familiar tool at a
higher level so we can see changes to nodes (names, inputs, outputs, tags).Installationpipinstallkedro-diffExamplekedro diff --stat develop..master
M  __default__      | 6 ++++-
M  data_science     | 3 +++
M  data_engineering | 3 ++-
?? new_pipeline

4 pipelines changed, 5 insertions(+), 4 deletions(-)Usage# diff develop into master
kedro diff develop..master

kedro diff develop master

# diff current state with main
kedro diff main

# diff current state with main
kedro diff ..main

# comparing pipelines from two branches
kedro diff master new_branch data_scienceMore exampleskedro diff develop..master
╭──────────────────────────────────────────────────────────────────────────────╮
│ modified: data_engineering                                                   │
╰──────────────────────────────────────────────────────────────────────────────╯+ strip_whitespace+ lowercase_columns+ get_trains- get_tains╭──────────────────────────────────────────────────────────────────────────────╮
│ modified: data_science                                                       │
╰──────────────────────────────────────────────────────────────────────────────╯+ split_dataRoadmap1.0.0commit parsergetpipeline.to_json()for__default__for two different commitsgetpipeline.to_json()for all pipelines for two different commits--stat compares the number of nodes added or dropped in__default__--stat compares the number of nodes added or dropped in all pipelines--stat compares attribute changes (inputs, outputs, tags) in all pipelinescompare input namescompare output namesspeed up getting repeat pipelines from the same commit (no need to reaload a new session)speed up getting repeat commits by checking commit hash (reuse existing json)minimize untested code2.0.0super-sizepipeline.to_json()compare all attributes on a node ( not just inputs, outputs, tags)allow users to specify custom to_json methodfunction namesfunction hashescatalog _filepathcatalog _sqlTestingThis project strives for 100% test coverage where it makes sense.  Other kedro
plugins I have created have suffered development speed by the complexity of
fully testing on a full kedro project. There are so many pieces to get into
place that it becomes difficult to test accross multiple versions of kedro or
keep the tests working as kedro changes.  Minimal functionality will be placed
into modules that require a kedro full kedro project in place to work.
Package: kedro-great
Kedro GreatAs Seen onDataEngineerOneWatch the Video:Kedro Great: Use Great Expectations with Ease!Kedro Great is an easy-to-use plugin for kedro that makes integration with Great Expectations fast and simple.Hold yourself accountable toGreat Expectations.Never have fear of data silently changing ever again.Quick StartInstallKedro Great is available on pypi, and is installed withkedro hooks.pip install kedro-greatSetupOnce installed,kedro greatbecomes available as a kedro command.You can usekedro great initto initialize a Great Expectations project, and then automatically generate its project context.Furthermore, by usingkedro great init, you also generate Great ExpectationsDatasources andSuites to use with yourcatalog.ymlDataSets.By default, expectation suites are named for thecatalog.ymlname and abasic.jsonis generated for each.kedro great initUseAfter the Great Expectations project has been setup and configured, you can now use theKedroGreathook to run all your data validations every time the pipeline runs.# run.pyfromkedro_greatimportKedroGreatclassProjectContext(KedroContext):hooks=(KedroGreat(),)Then just run the kedro pipeline to run the suites.kedro runResultsFinally, you can usegreat_expectationsitself to generate documentation and view the results of your pipeline.Love seeing those green ticks!great_expectations docs buildHook OptionsTheKedroGreathook supports a few options currently. If you wish toexpectations_map: Dict[str, Union[str, List[str]]]If you have multiple expectation suites you wish to run, or expectation suites that do not have the same name
as the catalog dataset, these mappings can be specified in theexpectations_mapargument forKedroGreatDefault:The catalog name is the expectation name.Note:Specifying a suite type such as.basicwill override all other suite typesKedroGreat(expectations_map={'pandas_iris_data':'pandas_iris_data','spark_iris_data':['spark_iris_data','other_expectation','another_expectation.basic'],})suite_types: List[Optional[str]]If your suites have multiple types, you can choose exactly which types to run.ANonemeans that a suite will not have the type appended to the name.Default:TheKedroGreat.DEFAULT_SUITE_TYPES.Node:If a suite type is already specified in theexpectations_map, that will override this list.KedroGreat(suite_types=['warning','basic',None])run_before_node:bool, run_after_node: boolYou can decide when the suites run, before or after a node or both before and after a node.It will operate on the nodeinputsandoutputsrespectively.Default:Only runs before a node runs.KedroGreat(run_before_node=True,run_after_node=False)fail_fast: bool, fail_after_pipeline_run: boolYou can also haveKedroGreatthrow aSuiteValidationFailurewhen a Great Expectations validation fails.Either the exception can be throw immediately, or the exceptions can be aggregated over the whole pipeline run, and thrown at the end.This is useful for when you wish to run validation on your pipeline in a CI/CD way.Default:Neither are setKedroGreat(fail_fast=True,fail_after_pipeline_run=True)
Package: kedro-grpc-server
Kedro gRPC ServerThis is a beta Kedro plugin that creates a gRPC server for your kedro pipelines. Kedro gRPC Server RPC calls can be triggered using any of the programming languages that support gRPC.Kedro gRPC clients can be in many programming languages.Some kedro client examples below that call RPCs on a gRPC server running in any kedro project:PythonGoScalaJavaNodeC++C#KotlinDartMore ongrpc.ioCode generation support in all gRPC frameworks makes it super easy to use for clients who may have a different tech stack and language preference. Simply sharing a kedro.proto file, clients can use their preferred programming language to generate strongly typed kedro clients in any of the languages mentioned above.Allowing engineers to natively integrate running of kedro pipelines with their preferred programming language.Getting status of kedro pipeline run as a streaming response through HTTP/2. gRPC provides first-class support for this.Getting all the benefits of gRPCAnd, enabling business users to interact with analytics from a front-end application and trigger actions or models (e.g. scoring  model) on demand.How do I install Kedro gRPC Server?PrerequisitesKedro gRPC Server requiresPython 3.6+andGitto be setup.InstallationYou can install Kedro gRPC Server directly from GitHub with:pipinstallgit+https://github.com/mmchougule/kedro-grpc-server.gitHow do I use Kedro gRPC Server?To start the server, simply run the following command inside your Kedro project:kedroservergrpc-startYou can specify the host through the flag like so:kedroservergrpc-start--hostSimilarly, you can set the port number using--port.RungRPC APIExposing 3 RPC calls similar to the REST server:ListPipelines-> Returns current list of pipelinesRun-> Runs a pipeline with or without argumentsStatus-> Provides run status of a pipeline with run_id.
The response for this rpc call is a Server Streaming response of all logged events.ContributingPlease readCONTRIBUTING.mdfor:The contribution code of conductThe process for submitting pull requestsVersioningWe useSemVerfor versioning. The best way to safely upgrade is to check ourrelease notesfor any notable breaking changes.What licence do you use?SeeLICENSEfor guidelines.
Package: kedro-hyperparam
FeaturesTODORequirementsTODOInstallationYou can installKedro HyperparamviapipfromPyPI:$pipinstallkedro-hyperparamUsagePlease see theCommand-line Referencefor details.ContributingContributions are very welcome.
To learn more, see theContributor Guide.LicenseDistributed under the terms of theMIT license,Kedro Hyperparamis free and open source software.IssuesIf you encounter any problems,
pleasefile an issuealong with a detailed description.CreditsThis project was generated from@cjolowicz’sHypermodern Python Cookiecuttertemplate.
Package: kedroio
kedroioA module extending the datasets that come shipped withkedroExample usage-- example.sqlselect*from"database"."table_name"limit5;# conf/base/catalog.pymy_athena_dataset:type:kedroio.datasets.aws.athena.AthenaQueryDataSetfilepath:data/01_raw/example.csvsql_filepath:example.sqlbucket:example-bucketworkgroup:primarysubfolder:dataregion_name:eu-west-2read_result:true# read into pandas DataFrameoverwrite:false# skip download if filepath existsTestingStartmotoserver for mocked AWS resourcesmoto_serverRun testspytesttests/
Package: kedro-kubeflow
Kedro Kubeflow PluginAboutThe main purpose of this plugin is to enable running kedro pipeline on Kubeflow Pipelines. It supports translation from
Kedro pipeline DSL tokfp(pipelines SDK) and deployment to
a running kubeflow cluster with some convenient commands.The plugin can be used together withkedro-dockerto simplify preparation of docker image for pipeline execution.DocumentationFor detailed documentation refer tohttps://kedro-kubeflow.readthedocs.io/Usage guideUsage: kedro kubeflow [OPTIONS] COMMAND [ARGS]...
 
   Interact with Kubeflow Pipelines
 
 Options:
   -h, --help  Show this message and exit.
 
 Commands:
   compile          Translates Kedro pipeline into YAML file with Kubeflow pipeline definition
   init             Initializes configuration for the plugin
   list-pipelines   List deployed pipeline definitions
   run-once         Deploy pipeline as a single run within given experiment.
   schedule         Schedules recurring execution of latest version of the pipeline
   ui               Open Kubeflow Pipelines UI in new browser tab
   upload-pipeline  Uploads pipeline to Kubeflow serverConfiguration filekedro initgenerates configuration file for the plugin, but users may want
to adjust it to match the run environment requirements:https://kedro-kubeflow.readthedocs.io/en/latest/source/02_installation/02_configuration.html
Package: kedro-local-notify
kedro-local-notifyHow do I get started?$pipinstall--upgradekedro-local-notifyThen what?Nothing! Kedro will automagically pick up the hook and ping you a notification after the pipeline finished running succesfully or fails.What iskedro-local-notify?Ever kicked off a long-running pipeline and come back to check in an hour later, only to find that it failed 2 minutes in?Or come back to see your pipeline finished running an hour ago and you have nothing to justify your reddit browsing anymore?kedro-local-notifywill ping you a notification indicating that your pipeline ran sucessfully or failed.Won't this spam me?By default, notifications will only trigger if the pipeline has been running for more than 1 minute. You can change this thresholf for notifying you by setting theKEDRO_LOCAL_NOTIFY_THRESHOLDenvironment variable to be the number of seconds of pipeline run time before a notification is trigerred. The default is:$exportKEDRO_LOCAL_NOTIFY=60note that this environment variable needs to be set in the same shell that you're trigerring the Kedro pipeline run in.CaveatsYou probably shouldn't add this to your requirements. It's a silly little tool meant to be used for some quality of life improvements on your local machine.This is currently limited to Mac OS X 10.10 or higher. Windows support is in the works!
Package: kedro-lsp
kedro-lspALanguage Serverfor the latest version(s) ofKedro. It provides features to enable IDE support for Kedro. For example, you can jump to dataset and parameter definition when constructing the pipeline.Note: This is pre-alpha software.FeaturesProvide dataset and parameter definition when constructing the pipeline.Note: I need your help! If you think this project is a good idea, please submit features request via Github Issue.CompatibilityKedro Language Server aims to be compatible with Kedro 0.17.x and above. Currently it is restricted to 0.17.3 and above during pre-alpha phase.Installationpipinstallkedro-lspUsageStandloneusage: kedro-lsp [-h] [--version] [--tcp] [--host HOST] [--port PORT] [--log-file LOG_FILE] [-v]

Kedro Language Server: an LSP wrapper for Kedro.

optional arguments:
  -h, --help           show this help message and exit
  --version            display version information and exit
  --tcp                use TCP server instead of stdio
  --host HOST          host for TCP server (default 127.0.0.1)
  --port PORT          port for TCP server (default 2087)
  --log-file LOG_FILE  redirect logs to the given file instead of writing to stderr
  -v, --verbose        increase verbosity of log output

Examples:
    Run from stdio: kedro-lspVisual Studio CodeTo use it with visual studio code, install the Kedro extension from Visual Studio Code Marketplace.PycharmTBDnvimTo use it with nvim, installlspconfig.:Plug 'neovim/nvim-lspconfig'Then add the following config to your vimrc.lua<<EOF
local configs=require'lspconfig/configs'configs.kedro={
    default_config={
        cmd={"kedro-lsp"};
        filetypes={"python"};
        root_dir=function(fname)returnvim.fn.getcwd()end;
    };
};
EOF🗒️ Note, you must have thekedro-lspinstalled and on yourPATHThere are a number of plugins that will help setup lsp functionality in nvim, but if you want a bare minimum go-to-definition add this to your.vimrcas well.nnoremap<leader>n<cmd>luavim.lsp.buf.definition()<CR>If you are having any issues withkedro-lspnot working in nvim try running:LspInfoor looking into your~/.cache/nvim/lsp.logfile.TodosProvide diagnostic when there is a typo in dataset or parameter name in the pipeline.Be Kedro environment-awareLicenseMIT
Package: kedro-neptune
Neptune + Kedro integrationKedro plugin for experiment tracking and metadata management. It lets you browse, filter and sort runs in a nice UI, visualize node/pipeline metadata, and compare pipelines.What will you get with this integration?browse, filter, and sortyour model training runscompare nodes and pipelineson metrics, visual node outputs, and moredisplay all pipeline metadataincluding learning curves for metrics, plots, and images, rich media like video and audio or interactive visualizations from Plotly, Altair, or Bokehand do whatever else you would expect from a modern ML metadata storeKedro pipeline metadata in custom dashboard in the Neptune web appResourcesDocumentationCode example on GitHubExample run logged in the Neptune appSupportIf you got stuck or simply want to talk to us, here are your options:Check ourFAQ pageYou can submit bug reports, feature requests, or contributions directly to the repository.Chat! When in the Neptune application click on the blue message icon in the bottom-right corner and send a message. A real person will talk to you ASAP (typically very ASAP),You can just shoot us an email atsupport@neptune.ai
Package: kedro-onnx
Kedro ONNXIntroductionAdds ONNX support to Kedro.UsageRead ourdocs.Getting StartedExecute this command in your terminal:pipinstallkedro-onnxContributeIf you want to contribute to this package, execute the following steps:Fork the repoClone it in your machineInstall withpip install .[dev]Runpre-commit installin the project rootMake your changesCreate a prNote: Remember to create unit tests and to run pytest and flake8 before creating a prCreditsNickolas da Rocha Machado@nickolasrm
Package: kedro-pack
kedro-packA generic recipe to install the kedro framework and additional kedro plugins.$pythonPython3.10.12|packagedbyconda-forge|(main,Jun232023,22:40:32)[GCC12.3.0]onlinuxType"help","copyright","credits"or"license"formoreinformation.>>>importkedro_pack>>>kedro_pack.get_kedro_component_versions()Thefollowingcomponentsareinstalledwithkedro-pack(0.18.12.1):-kedro:0.18.12-kedro-datasets:1.6.0-kedro-airflow:0.6.0-kedro-mlflow:0.11.9-kedro-viz:6.4.0-kedro-light:2023.3.13
Package: kedro-pandera
General informationsSoftware repositoryLatest releaseTotal downloadsPypiCode healthBranchTestsCoverageLinksDocumentationDeploymentActivitymainWhat is kedro-pandera?How do I install kedro-pandera?kedro-panderais available on PyPI, so you can install it withpip:pip install kedro-panderaIf you want to use the most up to date version of the package which is under development and not released yet, you can install the package from github:pip install --upgrade git+https://github.com/Galileo-Galilei/kedro-pandera.gitGetting startedThe documentation contains:A "hello world" exampleRelease and roadmapTherelease historycentralizes packages improvements across time. The main features coming in next releases arelisted on github milestones. Feel free to upvote/downvote and discuss prioritization in associated issues.DisclaimerThis package is still in active development. We useSemVerprinciples to version our releases. Until we reach1.0.0milestone, breaking changes will lead to<minor>version number increment, while releases which do not introduce breaking changes in the API will lead to<patch>version number increment.If you want to see how to migrate from one version ofkedro-panderato another, see themigration guide.Can I contribute?We'd be happy to receive help to maintain and improve the package. Any PR will be considered (from typo in the docs to core features add-on) Please check thecontributing guidelines.Main contributorsThe following people actively maintain, enhance and discuss design to make this package as good as possible:Yolan Honoré-RougéNok Lam ChanJoel Schwarzmann
Package: kedro-partitioned
Kedro PartitionedIntroductionKedro Plugin focused on extending and helping the user to process partitioned data.Usage:Read ourdocs.Getting StartedExecute this command in your terminal:pipinstallkedro-partitionedContributeIf you want to contribute with this package, do the following steps:Fork the repoClone it in your machineInstall withpip install .[dev]Runpre-commit installin the project rootMake your changesCreate a pr for us <3Note: Remember to create unit tests and to run pytest and flake8 before creating a prCreditsGabriel Daiha Alves@gabrieldaihaNickolas da Rocha Machado@nickolasrm
Package: kedro-popmon
Kedro PopmonKedro-popmon is a plugin to integrate popmon reporting with kedro.This plugin allows you to automate the process of popmon feature and output stability monitoring.How to installpipinstallkedro-popmonHow to use itRunkedro popmon initin a existing kedro project.You will get theconf/base/popmon.yml.Editconf/base/popmon.ymlaccording to the way described in it.Example:example_iris_data:
  date_column: date
  columns:
    - sepal_lengthWhen you dokedro runyou shall expect your reports to be generated indata/08_reporting/folder.
Package: kedro-projetaai
Kedro ProjetaAiIntroductionKedro ProjetaAi is a Kedro Plugin that discretizes production patterns, and
extends Kedro in order to enable cloud solutions into a Kedro environment.UsableCLI Extensionkedro_projetaai.cli.plugin.ProjetaAiCLIPluginprovides an interface with
predefined commands to inherit and override in order to create new integrations
between ProjetaAi/Kedro and a cloud solution.
Package: kedro-sagemaker
Kedro SageMaker Pipelines pluginWe help companies turn their data into assetsAboutThis plugin enables you to run Kedro projects on Amazon SageMaker. Simply install the package and use the providedkedro sagemakercommands to build, push, and run your project on SageMaker.DocumentationFor detailed documentation refer tohttps://kedro-sagemaker.readthedocs.io/Usage guideUsage: kedro sagemaker [OPTIONS] COMMAND [ARGS]...

Options:
  -e, --env TEXT  Environment to use.
  -h, --help      Show this message and exit.

Commands:
  compile  Compiles the pipeline to a JSON file
  init     Creates basic configuration for Kedro SageMaker plugin
  run      Runs the pipeline on SageMaker PipelinesQuickstartFollowquickstartsection onkedro-sagemaker.readthedocs.ioto see how to run your Kedro project on AWS SageMaker or watch the video below: