Package: afctl
afctlThe proposed CLI tool is authored to make creating and deployment of airflow projects faster and smoother.
As of now, there is no tool out there that can empower the user to create a boilerplate code structure for airflow
projects and make development + deployment of projects seamless.RequirementsPython 3.5+DockerGetting Started1. InstallationCreate a new python virtualenv. You can use the following command.python3-mvenv<name>Activate your virtualenvsource/path_to_venv/bin/activatepip3installafctl2. Initialize a new afctl project.The project is created in your present working directory. Along with this a configuration file with the same name is
generated in/home/.afctl_configsdirectory.afctlinit<nameoftheproject>Eg.afctlinitproject_demoThe following directory structure will be generated.
├──deployments
│└──project_demo-docker-compose.yml
├──migrations
├──plugins
├──project_demo
│├──commons
│└──dags
├──requirements.txt
└──testsIf you already have a git repository and want to turn it into an afctl project.
Run the following command :-afctlinit.3. Add a new module in the project.afctlgeneratemodule-n<nameofthemodule>The following directory structure will be generated :afctlgeneratemodule-nfirst_module
afctlgeneratemodule-nsecond_module

.
├──deployments
│└──project_demo-docker-compose.yml
├──migrations
├──plugins
├──project_demo
│├──commons
│└──dags
│├──first_module
│└──second_module
├──requirements.txt
└──tests├──first_module└──second_module4. Generate dagafctlgeneratedag-n<nameofdag>-m<nameofmodule>The following directory structure will be generate :afctlgeneratedag-nnew-mfirst_module

.
├──deployments
│└──project_demo-docker-compose.yml
├──migrations
├──plugins
├──project_demo
│├──commons
│└──dags
│├──first_module
││└──new_dag.py
│└──second_module
├──requirements.txt
└──tests├──first_module└──second_moduleThe dag file will look like this :fromairflowimportDAGfromdatetimeimportdatetime,timedeltadefault_args={'owner':'project_demo',# 'depends_on_past': ,# 'start_date': ,# 'email': ,# 'email_on_failure': ,# 'email_on_retry': ,# 'retries': 0}dag=DAG(dag_id='new',default_args=default_args,schedule_interval='@once')5. Deploy project locallyYou can add python packages that will be required by your dags inrequirements.txt. They will automatically get
installed.To deploy your project, run the following command (make sure docker is running) :afctldeploylocalIf you do not want to see the logs, you can runafctldeploylocal-dThis will run it in detached mode and won't print the logs on the console.You can access your airflow webserver on browser atlocalhost:80806. Deploy project on productionHere we will be deploying our project toQubole. Sign up at us.qubole.com.add git-origin and access-token (if want to keep the project as private repo
on Github) to the configs.See howPush the project once completed to Github.Deploying to Qubole will require adding deployment configurations.afctlconfigadd-dqubole-n<nameofdeployment>-e<env>-c<cluster-label>-t<auth-token>This command will modify your config file. You can see your config file with the following command :afctlconfigshowFor example -afctlconfigadd-dqubole-ndemo-ehttps://api.qubole.com-cairflow_1102-tkhd34djs3To deploy run the following commandafctldeployqubole-n<name>The following video also contains all the steps of deploying project using afctl -https://www.youtube.com/watch?v=A4rcZDGtJME&feature=youtu.beManage configurationsThe configuration file is used for deployment contains the following information.global:-airflow_version:-git:--origin:--access-token:deployment:-qubole:--local:---compose:airflow_versioncan be added to the project when you initialize the project.afctlinit<name>-v<version>global configs (airflow_version, origin, access-token) can all be added/ updated with the following command :afctlconfigglobal-o<git-origin>-t<access-token>-v<airflow_version>UsageCommands right now supported areinitconfigdeploylistgenerateTo learn more, runafctl<command>-hCautionNot yet ported for Windows.CreditsDocker-compose file :https://github.com/puckel/docker-airflow