Package: airflow-ecs-fargate-executor
AWS ECS and Fargate ExecutorThis is an AWS Executor that delegates every task to a scheduled container on either AWS ECS or AWS Fargate. By default, AWS Fargate will let you run
2000 simultaneous containers, with each container representing 1 Airflow Task.pipinstallairflow-ecs-fargate-executorGetting StartedIn your$AIRFLOW_HOME/pluginsfolder create a file calledecs_fargate_plugin.py.fromairflow.plugins_managerimportAirflowPluginfromairflow_ecs_fargate_executorimportEcsFargateExecutorclassEcsFargatePlugin(AirflowPlugin):"""AWS ECS & AWS FARGATE Plugin"""name="aws_ecs_plugin"executors=[EcsFargateExecutor]For more information on any of these execution parameters, see the link below:https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/ecs.html#ECS.Client.run_taskFor boto3 credential management, seehttps://boto3.amazonaws.com/v1/documentation/api/latest/guide/configuration.htmlHow It WorksEverytime Apache Airflow wants to run a task, this executor will use Boto3'sECS.run_task()function to schedule a container on an existing cluster. Then every scheduler heartbeat the executor will check the status of every running container, and sync it with Airflow.But Why?Pay for what you use. With Celery, there is no predefined concept of auto-scaling. Therefore the number of worker servers one must constantly provision, pay for, and maintain is a static number. Due to the sporadic nature of batch-processing, most of the time most of these servers are not in use. However, during peak hours, these servers become overloaded.Servers require up-keep and maintenance. For example, just one cpu-bound or memory-bound Airflow Task could overload the resources of a server and starve out the celery or scheduler thread;thus causing the entire server to go down. This executor mitigates this risk.Simplicity in Setup.No new libaries are introducedIf you're on the Fargate executor it may take 4 minutes for a task to pop up, but at least it's a contant number.  This way, the concept of tracking DAG Landing Times becomes unneccessary. If you have more than 2000 concurrent tasks (which is a lot) then you can always contact AWS to provide an increase in this soft-limit.AWS ECS v AWS Fargate?AWS Fargate- Is a serverless container orchistration service; comparable to a proprietary AWS version of Kubernetes. Launching a Fargate Task is like saying "I want these containers to be launched somewhere in the cloud with X CPU and Y memory, and I don't care about the server". AWS Fargate is built on top of AWS ECS, and is easier to manage and maintain. However, it provides less flexibility.AWS ECS- Is known as "Elastic Container Service", which is a container orchistration service that uses a designated cluster of EC2 instances that you operate, own, and maintain.I almost always recommend that you go the AWS Fargate route unless you need the custom flexibility provided by ECS.ECSFargateStart-up per taskInstantaneous if capacity available2-4 minutes per task; O(1) constant timeMaintenanceYou patch the own, operate, and patchServerlessCapacityDepends on number of machines with available space in cluster~2000 containers. See AWS LimitsFlexibilityHighLowAirflow Configurations[ecs_fargate]regiondescription: The name of AWS Regionmandatory: even with a custom run_task_templateexample: us-east-1clusterdescription: Name of AWS ECS or Fargate clustermandatory: even with a custom run_task_templatecontainer_namedescription: Name of registered Airflow container within your AWS cluster. This container will
receive an airflow CLI command as an additional parameter to its entrypoint.
For more info see url to Boto3 docs above.mandatory: even with a custom run_task_templatetask_definitiondescription: Name of AWS Task Definition. For more info see url to Boto3 docs above.launch_typedescription: Launch type can either be 'FARGATE' OR 'EC2'. For more info see url to Boto3 docs above.default: FARGATEplatform_versiondescription: AWS Fargate is versioned.See this page for more detailsdefault: LATESTassign_public_ipdescription: Assign public ip. For more info see url to Boto3 docs above.security_groupsdescription: Security group ids for task to run in (comma-separated). For more info see url to Boto3 docs above.subnetsdescription: Subnets for task to run in (comma-separated). For more info see url to Boto3 docs above.example: subnet-XXXXXXXX,subnet-YYYYYYYYrun_task_templatedescription: This is the default configuration for calling the ECSrun_taskfunction API (see url above).
To change the parameters used to run a task in FARGATE or ECS, the user can overwrite the path to
specify another jinja-templated JSON. More documentation can be found in theExtensibilitysection below.mandatory: even with a custom run_task_templatedefault: default_aws_ecs.DEFAULT_AWS_ECS_CONFIGNOTE: Modify airflow.cfg or export environmental variables. For example:AIRFLOW__ECS_FARGATE__REGION="us-west-2"ExtensibilityThere are many different ways to run an ECS or Fargate Container. You may want specific container overrides, environmental variables, subnets, etc. This project does not attempt to wrap around the AWS API. Instead, it allows the user to offer their own configuration in the form of Python dictionary, which are then passed in to Boto3's run_task function as **kwargs.In this example we will modify the DEFAULT_AWS_ECS_CONFIG. Note, however, there is nothing that's stopping us from complete overriding it and providing our own config. If we do so, the only manditory Airflow Configurations areregion,cluster,container_name, andrun_task_template.For example:exportAIRFLOW__AWS_ECS__RUN_TASK_TEMPLATE="aws_ecs_configs.AWS_ECS_CONFIG"# filename: AIRFLOW_HOME/plugins/aws_ecs_config.pyfromaws_ecs_default_configsimportDEFAULT_AWS_ECS_CONFIG# Add environmental variables to contianer overridesAWS_ECS_CONFIG=DEFAULT_AWS_ECS_CONFIGAWS_ECS_CONFIG['overrides']['containerOverrides'][0]['environment']=['SOME_ENV_A','SOME_ENV_B']Custom Container RequirementsThis means that you can specify CPU, Memory, and GPU requirements on a task.task=PythonOperator(python_callable=lambda*args,**kwargs:print('hello world'),task_id='say_hello',executor_config=dict(cpu=256,memory=512),dag=dag)