Package: kestra
Event-Driven Declarative Orchestrator"Click on the image to get started in 4 minutes with Kestra."Live DemoTry Kestra using ourlive demo.What is KestraKestra is an open-source,event-drivenorchestrator that simplifies data operations and improves collaboration between engineers and business users. By bringingInfrastructure as Codebest practices to data pipelines, Kestra allows you to build reliable workflows and manage them with confidence.Thanks to thedeclarative YAML interfacefor defining orchestration logic, everyone who benefits from analytics can participate in the data pipeline creation process. The UI automatically adjusts the YAML definition any time you make changes to a workflow from the UI or via an API call. Therefore, the orchestration logic is defined declaratively in code, even if some workflow components are modified in other ways.Key conceptsFlowis the main component in Kestra. It's a container for your tasks and orchestration logic.Namespaceis used to provide logical isolation, e.g., to separate development and production environments. Namespaces are like folders on your file system — they organize flows into logical categories and can be nested to provide a hierarchical structure.Tasksare atomic actions in a flow. By default, all tasks in the list will be executed sequentially, with additional customization options, a.o. to run tasks in parallel or allow a failure of specific tasks when needed.Triggersdefine when a flow should run. In Kestra, flows are triggered based on events. Examples of such events include:a regular time-basedscheduleanAPIcall (webhook trigger)ad-hoc execution from theUIaflow trigger- flows can be triggered from other flows using aflow triggeror asubflow, enabling highly modular workflows.custom events, including a new file arrival (file detection event), a new message in a message bus, query completion, and more.Inputsallow you to pass runtime-specific variables to a flow. They are strongly typed, and allow additionalvalidation rules.Extensible platform via pluginsMost tasks in Kestra are available asplugins, but many type of tasks are available in the core library, including a.o. script tasks supporting various programming languages (e.g., Python, Node, Bash) and the ability to orchestrate your business logic packaged into Docker container images.To create your own plugins, check theplugin developer guide.Rich orchestration capabilitiesKestra provides a variety of tasks to handle both simple and complex business logic, including:retriestimeouterror handlingconditional branchingdynamic taskssequential and parallel tasksskipping tasks or triggers when needed by setting the flagdisabledtotrue.configuring dependencies between tasks, flows and triggersadvanced scheduling and trigger conditionsbackfillsdocumenting your flows, tasks and triggers by adding a markdown description to any componentadding labels to add additional metadata to your flows such as the flow owner or team:id:hellonamespace:proddescription:Hi from `Kestra` and a **markdown** description.labels:owner:john-doeteam:data-engineeringtasks:-id:hellotype:io.kestra.core.tasks.log.Logmessage:Hello world!description:a *very* important taskdisabled:falsetimeout:10Mretry:type:constant# type: stringinterval:PT15M# type: DurationmaxDuration:PT1H# type: DurationmaxAttempt:5# type: intwarningOnRetry:true# type: boolean, default is false-id:paralleltype:io.kestra.core.tasks.flows.Parallelconcurrent:3tasks:-id:task1type:io.kestra.plugin.scripts.shell.Commandscommands:-'echo"running{{task.id}}"'-'sleep10'-id:task2type:io.kestra.plugin.scripts.shell.Commandscommands:-'echo"running{{task.id}}"'-'sleep10'-id:task3type:io.kestra.plugin.scripts.shell.Commandscommands:-'echo"running{{task.id}}"'-'sleep10'triggers:-id:scheduletype:io.kestra.core.models.triggers.types.Schedulecron:"*/15****"backfill:start:2023-06-25T14:00:00ZBuilt-in code editorYou can write workflows directly from the UI. When writing your workflows, the UI provides:autocompletionsyntax validationembedded plugin documentationtopology view (view of your dependencies in a Directed Acyclic Graph) that get updated live as you modify and add new tasks.Getting StartedTo get a local copy up and running, follow the steps below.PrerequisitesMake sure that Docker is installed and running on your system. The default installation requires the following:DockerDocker ComposeLaunch KestraDownload the Docker Compose file:curl-odocker-compose.ymlhttps://raw.githubusercontent.com/kestra-io/kestra/develop/docker-compose.ymlAlternatively, you can usewget https://raw.githubusercontent.com/kestra-io/kestra/develop/docker-compose.yml.Start Kestra:docker-composeupOpenhttp://localhost:8080in your browser and create your first flow.Hello-World flowHere is a simple example logging hello world message to the terminal:id:hellonamespace:prodtasks:-id:hello-worldtype:io.kestra.core.tasks.log.Logmessage:Hello world!For more information:Follow thegetting started tutorial.Read thedocumentationto understand how to:Develop your flowsDeploy KestraUse ourTerraform providerto deploy your flowsDevelop yourown plugins.PluginsKestra is built on aplugin system. You can find your plugin to interact with your provider; alternatively, you can followthese stepsto develop your own plugin.For a full list of plugins, check theplugins page.Here are some examples of the available plugins:Airbyte CloudAirbyte OSSAmazon AthenaAmazon CLIAmazon DynamoDbAmazon RedshiftAmazon S3Amazon SNSAmazon SQSAMQPApache AvroApache CassandraApache KafkaApache PinotApache ParquetApache PulsarApache SparkApache TikaAzure BatchAzure Blob StorageAzure Blob TableCSVClickHouseCompressionCouchbaseDatabricksdbt clouddbt coreDebezium Microsoft SQL ServerDebezium MYSQLDebezium PostgresDuckDbElasticSearchEmailFivetranFTPFTPSGitGoogle Big QueryGoogle Pub/SubGoogle Cloud StorageGoogle DataProcGoogle FirestoreGoogle CliGoogle Vertex AIGoogle Kubernetes EnginesGoogle DriveGoogle SheetsGroovyHttpJSONJuliaJythonKubernetesMicrosoft SQL ServerMicrosoft TeamsMongoDbMQTTMySQLNashornNATSNeo4jNodeOpenAIOpen PGPOraclePostgreSQLPower BIPowerShellPythonRocksetRScriptSFTPServiceNowSingerShellSlackSnowflakeSodaSSHTelegramTrinoXMLVerticaThis list is growing quickly and we welcome contributions.Community SupportIf you need help or have any questions, reach out using one of the following channels:Slack- join the community and get the latest updates.GitHub discussions- useful to start a conversation that is not a bug or feature request.Twitter- to follow up with the latest updates.RoadmapSee theopen issuesfor a list of proposed features (and known issues) or look at theproject board.ContributingWe love contributions, big or small. Check outour contributor guidefor details on how to contribute to Kestra.See ourPlugin Developer Guidefor details on developing and publishing Kestra plugins.LicenseApache 2.0 ©Kestra Technologies