Package: typhoon-orchestrator
️Website :loudspeaker:|Discord :sunglasses:|Forum :wave:|Installation :floppy_disk:|Documentation :notebook:Why Typhoon?Our vision is a new generation of cloud native, asynchronous orchestrators that can handle highly dynamic workflows with ease. We crafted Typhoon from the ground up to work towards this vision. It's designed to feel familiar while still making very different design decisions where it matters.Typhoon + AWS LambdaA Serverless orchestrator has the potential to be infinitely scalable and extremely cost efficient at the same time. We think AWS Lambda is ideal for this:CloudWatch Events can trigger a Lambda on a schedule, so we get scheduling for free! A scheduler is the most complex piece of an orchestrator. We can do away with it completely and still be sure that our DAGs will always run on time.Lambda is cheap. You get 1 million invocations for free every month.Workflows can be paralellized by running tasks in parallel on different instances of the Lambda. Typhoon DAGs use batching to take full advantage of this.Typhoon + AirflowAirflow is great!It's also the industry standard and will be around for a while. However, we think it can be improved, without even migrating your existing production code.Typhoon lets you write Airflow DAGS faster:rocket::**Workflow**: Typhoon YAML DAG --> Typhoon build --> Airflow DAGSimplicity and re-usability; a toolkit designed to be loved by Data Engineers :heart:Key featuresPure python- Easily extend with pure python. Frameworkless, with no dependencies.Testable Python- Write tests for your tasks in PyTest. Automate DAG testing.Composability- Functions and connections combine like Lego. Very easy to extend.Data sharing- data flows between tasks making it intuitive to build tasks.Elegant: YAML- low-code and easy to learn.Code-completion- Fast to compose. (VS Code recommended).Components- reduce complex tasks (e.g. CSV → S3 → Snowflake) to 1 re-usable task.Components UI-  Share your pre-built automation with your team. teams. :raised_hands:Rich Cli & Shell- Inspired by other great command line interfaces and instantly familiar. Intelligent bash/zsh completion.Flexible deployment- Deploy to Airflow. Large reduction in effort, without breaking existing production.Example YAML DAGname:favorite_authorsschedule_interval:rate(1 day)tasks:choose_favorites:function:typhoon.flow_control.branchargs:branches:-J. K. Rowling-George R. R. Martin-James Clavellget_author:input:choose_favoritesfunction:functions.open_library_api.get_authorargs:author:!Py$BATCHwrite_author_json:input:get_authorfunction:typhoon.filesystem.write_dataargs:hook:!Hookdata_lakedata:!MultiStep-!Py$BATCH['docs']-!Pytyphoon.data.json_array_to_json_records($1)path:!MultiStep-!Py$BATCH['docs'][0]['key']-!Pyf'/authors/{$1}.json'create_intermediate_dirs:TrueGetting the works of my favorite authors from Open Library API⚡ InstallationSeedocumentationfor more extensive installation instructions and walkthroughs.with pip (typhoon standalone)Install typhoon:pipinstalltyphoon-orchestrator[dev]# Create a projecttyphooninithello_world# Try the Clicdhello_world
typhoonstatus# Add your connectiontyphoonconnectionadd--conn-iddata_lake--conn-envlocaltyphoonconnectionls-lDocs:Detailed local installation instructions.  |Hello world.With Docker and AirflowTo deploy Typhoon with Airflow you need:Docker / Docker Desktop (For now,you must use Gitbash on Windows. Currently, there is an open issue onWSL2.)Download the [docker-compose.yaml][1]  (or use curl below)Create a directory for your TYPHOON_PROJECTS_HOMEThe following sets up your project directory and gets the docker-compose.yml:TYPHOON_PROJECTS_HOME="/tmp/typhoon_projects"# Or any other path you prefermkdir-p$TYPHOON_PROJECTS_HOME/typhoon_airflow_testcd$TYPHOON_PROJECTS_HOME/typhoon_airflow_test# For Windows WSL2 Users - for other env. its optionalsudochown-R$USER:$TYPHOON_PROJECTS_HOME/typhoon_airflow_test
mkdirairflow
mkdirdata_lake
mkdirsrc

curl-LfOhttps://raw.githubusercontent.com/typhoon-data-org/typhoon-orchestrator/master/docker-compose-af.yml!!! Important
On WindowsGitbashplease run each docker-compose runone by one. They are quick.docker-compose-fdocker-compose-af.ymlrun--rmtyphoon-afairflowinitdb
docker-compose-fdocker-compose-af.ymlrun--rmtyphoon-aftyphoonstatus
docker-compose-fdocker-compose-af.ymlrun--rmtyphoon-aftyphoonconnectionadd--conn-iddata_lake--conn-envlocal# Adding our first connection!docker-compose-fdocker-compose-af.ymlrun--rmtyphoon-aftyphoondagbuild--all
dockercompose-fdocker-compose-af.ymlup-dThis runs a container with only 1 service,typhoon-af. This has both Airflow and Typhoon installed on it ready to work with.You should be able to then checktyphoon statusand also the airlfow UI athttp://localhost:8088Docs:Detailed docker installation instructions.  |Development hints.![Airflow UI](docs/img/airflow_ui_list_after_install.png)
*Typhoon DAGS listed in airflow UI*Favorite Authors DAG - as displayed in airflow UI