Package: gusty
gusty allows you to control your Airflow DAGs, Task Groups, and Tasks with greater ease. gusty manages collections of tasks, represented as any number of YAML, Python, SQL, Jupyter Notebook, or R Markdown files. A directory of task files is instantly rendered into a DAG by passing a file path to gusty'screate_dagfunction.gusty also manages dependencies (within one DAG) and external dependencies (dependencies on tasks in other DAGs) for each task file you define. All you have to do is provide a list ofdependenciesorexternal_dependenciesinside of a task file, and gusty will automatically set each task's dependencies and create external task sensors for any external dependencies listed.gusty works with both Airflow 1.x and Airflow 2.x, and has even more features, all of which aim to make the creation, management, and iteration of DAGs more fluid, so that you can intuitively design your DAG and build your tasks.DocumentationThe official documentation for gusty is hosted here:https://pipeline-tools.github.io/gusty-docs/What's in gusty?Five Ways to Make Tasksgusty will turn every file in a DAG directory into a task. By default gusty supports five different file types, which offer convenient ways to specify an operator and operator parameters for task creation.File TypeHow It Works.ymlDeclare anoperatorand pass in any operator parameters using YAML.pySimply write Python code and by default gusty will execute your file using aPythonOperator. Other options available.sqlDeclare anoperatorin a YAML header, then write SQL in the main .sql file. The SQL automatically gets sent to the operator.ipynbPut a YAML block at the top of your notebook and specify anoperatorthat renders your Jupyter Notebook.RmdUse the YAML block at the top of your notebook and specify anoperatorthat renders your R Markdown DocumentHere is quick example of a YAML task file, which might be called something likehello_world.yml:operator: airflow.operators.bash.BashOperator
bash_command: echo hello worldThe resulting task would be aBashOperatorwith the task idhello_world.Here is the same approach using a Python file instead, namedhello_world.py, which gusty will automatically turn into aPythonOperatorby default:phrase="hello world"print(phrase)Lastly, here's a slightly different.sqlexample:---operator:airflow.providers.sqlite.operators.sqlite.SqliteOperator---SELECTcolumn_1,column_2FROMyour_tableEasy DependenciesDeclarative DependenciesEvery task file type supportsdependenciesandexternal_dependenciesparameters, which gusty will use to automatically assign dependencies between tasks and create external task sensors for any external dependencies listed for a given task.For .yml, .ipynb, and .Rmd task file types, dependencies and external_dependencies would be defined using YAML syntax:operator: airflow.operators.bash.BashOperator
bash_command: echo hello world
dependencies:
  - same_dag_task
external_dependencies:
  - another_dag: another_task
  - a_whole_dag: allFor external dependencies, the keywordallcan be used when the task should wait on an entire external DAG to run successfully.For a .py task file type, we can define these dependencies with some raw markdown at the top of the file:# ---# dependencies:#   - same_dag_task# external_dependencies:#   - another_dag: another_task#   - a_whole_dag: all# python_callable: say_hello# ---defsay_hello():phrase="hello world"print(phrase)You will also note that we wrapped our previous Python code in a function calledsay_hello, and passed this function's name to thepython_callableargument. By default, with nooperatorand nopython_callablespecified, gusty will pass a simple function that runs your .py file to thePythonOperator. If you pass an explicitpython_callableby name, gusty will search your .py for that function and pass that function to thePythonOperatorinstead..py files are capable of accepting anoperatorparameter in the raw markdown, just like any other task file type, which means you can use any other relevant operators (e.g. thePythonVirtualenvOperator) to execute your Python code as needed.Dynamic Dependenciesgusty can also detect and generate dependencies through a task object'sdependenciesattribute. This means you can alsodynamicallyset dependencies. One popular example of this option would be if your operator runs SQL, you can parse that SQL for table names, and attach a list of those table names to the operator'sdependenciesattribute. If those table names listed in thedependenciesattribute are also task ids in the DAG, gusty will be able to automatically set these dependencies for you!DAG and TaskGroup ControlBoth DAG and TaskGroup objects are created automatically simply by being directories and subfolders, respectively. The directory path you provide to gusty'screate_dagfunction will become your DAG (and DAG name), and any subfolder in that DAG by default will be turned into a TaskGroup.gusty offers a few compatible methods for configuring DAGs and Task Groups that we'll cover below.MetadataA special file name in any directory or subfolder isMETADATA.yml, which gusty will use to determine how to configure that DAG or TaskGroup object.Here is an example of aMETADATA.ymlfile you might place in a DAG directory:description: "An example of a DAG created using METADATA.yml"
schedule_interval: "1 0 * * *"
default_args:
    owner: airflow
    depends_on_past: False
    start_date: !days_ago 1
    email: airflow@example.com
    email_on_failure: False
    email_on_retry: False
    retries: 1
    retry_delay: !timedelta 'minutes: 5'And here is an example of aMETADATA.yml file you might place in a TaskGroup subfolder:tooltip: "This is a task group tooltip"
prefix_group_id: True
dependencies:
  - hello_worldAs seen in the above example, gusty will also acceptdependenciesandexternal_dependenciesin a TaskGroup'sMETADATA.yml. This means gusty can wire up your TaskGroup dependencies as well!Note that gusty disables the TaskGroupprefix_group_idargument by default, as it's one of gusty's few opinions that tasks should explicitly named unless you say otherwise. gusty also offers asuffix_group_idargument for Task Groups!create_dagWhileMETADATA.ymlwill always be the primary source of truth for a DAG or TaskGroup's configuration, gusty'screate_dagfunction also accepts any parameters that can be passed to Airflow's DAG class, as well as a dictionary oftask_group_defaultsto set default behavior for any TaskGroup created by gusty.Here's an example usingcreate_dag, where instead of metadata we usecreate_dagarguments:fromdatetimeimporttimedeltafromairflow.utils.datesimportdays_agofromgustyimportcreate_dagdag=create_dag('/usr/local/airflow/dags/hello_world',description="A DAG created without any metadata",schedule_interval="1 0 * * *",default_args={"owner":"airflow","depends_on_past":False,"start_date":days_ago(1),"email":"airflow@example.com","email_on_failure":False,"email_on_retry":False,"retries":1,"retry_delay":timedelta(minutes=5),},task_group_defaults={"tooltip":"This is a task group tooltip","prefix_group_id":True})You might notice thattask_group_defaultsdoes not include dependencies. For Task Groups, dependencies must be set using TaskGroup-specific metadata.Default arguments increate_dagand a DAG or TaskGroup'sMETADATA.ymlcan be mixed and matched.METADATA.ymlwill always override defaults set increate_dag.create_dagsIf you have multiple gusty DAGs located inside of a single directory, you can conveniently use thecreate_dags(plural) function.create_dagsworks just likecreate_dag, with two exceptions:The first argument tocreate_dagsis the path to a directory with many gusty DAGs.The second argument tocreate_dagsisglobals().globals()is essentially the namespace to which your DAGs are assigned.Let's adjust the abovecreate_dagexample to usecreate_dagsinstead:fromdatetimeimporttimedeltafromairflow.utils.datesimportdays_agofromgustyimportcreate_dagscreate_dags('/usr/local/airflow/my_gusty_dags',globals(),description="A default description for my DAGs.",schedule_interval="1 0 * * *",default_args={"owner":"airflow","depends_on_past":False,"start_date":days_ago(1),"email":"airflow@example.com","email_on_failure":False,"email_on_retry":False,"retries":1,"retry_delay":timedelta(minutes=5),},task_group_defaults={"tooltip":"This is a task group tooltip","prefix_group_id":True})The above will create many gusty DAGs located in the/usr/local/airflow/my_gusty_dagsdirectory.DAG-level Featuresgusty features additional helpful arguments at the DAG-level to help you design your DAGs with ease:root_tasks- A list of task ids which should represent the roots of a DAG. For example, an HTTP sensor might have to succeed before any downstream tasks in the DAG run.leaf_tasks- A list of task ids which should represent the leaves of a DAG. For example, at the end of the DAG run, you might save a report to S3.external_dependencies- You can also set external dependencies at the DAG level! Making your DAG wait on other DAGs works just like in the external dependencies examples above.ignore_subfolders- If you don't want subfolders to generate Task Groups, set this toTrue.latest_only- On by default, installs aLatestOnlyOperatorat the absolute root of the DAG, skipping all tasks in the DAG if the DAG run is not the current run. You can read more about the LatestOnlyOperator inAirflow's documentation.Any of these arguments can be placed increate_dagorMETADATA.yml!Local Operator SupportWhile you can store your local operators in Airflow'spluginsdirectory and reference an operator'spluginspath accordingly, gusty also allows you to alternatively keep local operators inside of anoperatorsfolder located inside of yourAIRFLOW_HOME.In order for gusty to support your operators as expected, your operator name must be CamelCase and the file in which the operator lives must be snake_case.For example, if we wanted use aHelloOperator, this operator would need to be stored in a file calledhello_operator.pyin inside of theoperatorsfolder located inside of yourAIRFLOW_HOME.Any fields from your operator's__init__method will be passed from gusty to your operator. So if yourHelloOperatorhad anamefield, you could call this operator with a YAML task file that looks something like this:operator: local.HelloOperator
name: WorldThelocal.syntax is what gusty uses to know to look in your local operators folder for the operator.Multi-Task GenerationSometimes task definitions can be repetitive. To account for this, gusty allows for amulti_task_specblock in any frontmatter. This allows you to generate multiple similar tasks with a single task definition file! For example, let's say you wanted to create two bash tasks, each containing a differentbash_command. You can define these two tasks in a single task definition file like so:operator: airflow.operators.bash.BashOperator
multi_task_spec:
  bash_task_1:
    bash_command: echo first_task
  bash_task_2:
    bash_command: echo second_taskgusty will convert the above into two task instances,bash_task_1andbash_task_2, each with a uniquebash_command.Additionally, for the special case ofpython_callablein a .py file, you can specifypython_callable_partials:# ---# python_callable: main# python_callable_partials:#   python_task_1:#       my_kwarg: a#   python_task_2:#       my_kwarg: b# ---defmain(my_kwarg):returnmy_kwarggusty will convert the above in two task instances,python_task_1andpython_task_2.python_task_1will return"a"andpython_task_2will return"b".multi_task_specandpython_callable_partialsare non-exclusive, so you can mix and match configuration as needed.One Approach, Not the Only ApproachOne good thing about gusty is that if you choose to use this package, gusty doesn't have to be theonlyway that you create DAGs. You can use gusty'screate_dagto generate DAGs out of directories where applicable, and then implement more traditional methods of creating Airflow DAGs where the tried and true methods feel like a better approach.So feel free to give the gusty approach a try, because you don't have to commit to it everywhere. But when you try it, don't be surprised if you start using it everywhere!Containerized DemoAs an additional resource, you can check out a containerized demo of gusty and Airflow over at thegusty-demo repo, which illustrates how gusty and a few custom operators can make SQL queries, Jupyter notebooks, and RMarkdown documents all work together in the same data pipeline.DevelopmentThe below assumes you haveDockerinstalled.Starting for the First TimeFirst,git clonethis repository.Then:exportGUSTY_DEV_HOME="~/path/to/this/project"cd$GUSTY_DEV_HOMEmakebuild-image
makerun-imageThe above will build the development image under the namegusty-testingand run a container calledgusty-testing.From here, you can:make exec- Exec into a terminal in the running container.make test- Runspytestin a temporary container.make coverage- Runspytestand generates a coverage report.make browse-coverage- Opens up the coverage report in your browser.make stop-container- Stop the running container.make start-container- Start a stopped container.Rebuilding the Imagemakestop-container# if you have a running containermakeremove-container# if you have a stopped containermakebuild-image
makerun-image