Package: awsdp
AWS Data PlatformThis repo contains the code for building a data platform on AWS.We enable 2 data environments:dev: A development env running on dockerprd: A production env running on aws + k8sSetupCreate + activate a virtual env:python3-mvenv~/.venvs/dpenvsource~/.venvs/dpenv/bin/activateInstall + initphidata:pipinstallphidata
phiinitfrom thedata-platformdir:Setup workspace:phiwssetupCopyworkspace/example_secretstoworkspace/secrets:cp-rworkspace/example_secretsworkspace/secretsDeploy dev containers to docker using:phiwsupphiwill create the following resources:Container:dev-pg-dp-containerNetwork:dpOptional: If something fails, try running again with debug logs:phiwsup-dOptional: Create.envfile:cpexample.env.envUsing the dev environmentTheworkspace/devdirectory contains the code for the dev resources. Theworkspace/settings.pyfile can be used to enable the open-source applications like:Postgres App: for storing dev data (runs 1 container)Airflow App: for running dags & pipelines (runs 5 containers)Superset App: for visualizing dev data (runs 4 containers)Update theworkspace/settings.pyfile and run:phiwsupTIP: Thephi ws ...commands use--env devand--config dockerby default. Set in theworkspace/config.pyfile.Runningphi ws upis equivalent to runningphi ws up --env dev --config dockerRun AirflowSetairflow_enabled = Trueinworkspace/settings.pyand runphi ws upCheck out the airflow webserver running in theairflow-ws-container:url:http://localhost:8310/user:adminpass:adminSuperset webserverSetsuperset_enabled = Trueinworkspace/settings.pyand runphi ws upCheck out the superset webserver running in thesuperset-ws-container:url:http://localhost:8410/user:adminpass:adminFormat + lint workspaceFormat withblack& lint withmypyusing:./scripts/format.shIf you need to install packages, run:pipinstallblackmypyUpgrading phidata versionactivate virtualenv:source ~/.venvs/dpenv/bin/activateUpgrade phidata:pipinstallphidata--upgradeRebuild local images & recreate containers:CACHE=fphiwsup--envdev--configdockerOptional: Install workspace locallyInstall the workspace & python packages locally in your virtual env using:./scripts/install.shThis will:Install python packages fromrequirements.txtInstall python project in--editablemodeInstallrequirements-airflow.txtwithout dependencies for code completionThis enables:Runningblack&mypylocallyRunning workflows locallyEditor auto-completionAdd python packagesFollowingPEP-631, we should add dependencies to thepyproject.tomlfile.To add a new package:Add the module to thepyproject.tomlfile.Run:./scripts/upgrade.sh. This script updates therequirements.txtfile.Optional: Run:./scripts/install.shto install the new dependencies in a local virtual env.RunCACHE=f phi ws upto recreate images + containersAdding airflow providersAirflow requirements are stored in theworkspace/dev/airflow_resources/requirements-airflow.txtfile.To add new airflow providers:Add the module to theworkspace/dev/airflow_resources/requirements-airflow.txtfile.Optional: Run:./scripts/install.shto install the new dependencies in a local virtual env.RunCACHE=f phi ws up --name airflowto recreate images + containersTo force recreate all images & containers, use theCACHEenv variableCACHE=falsephiwsup\--envdev\--configdocker\--typeimage|container\--nameairflow|superset|pg\--appairflow|supersetShut down workspacephiwsdownRestart all resourcesphiwsrestartRestart all containersphiwsrestart--typecontainerRestart traefik appphiwsrestart--apptraefikRestart airflow appphiwsrestart--appairflowAdd environment/secret variables to your appsThe containers read env using theenv_fileand secrets using thesecrets_fileparams.
These files are stored in theworkspace/envorworkspace/secretsdirectories.AirflowTo add env variables to your airflow containers:Update theworkspace/env/dev_airflow_env.ymlfile.Restart all airflow containers using:phi ws restart --name airflow --type containerTo add secret variables to your airflow containers:Update theworkspace/secrets/dev_airflow_secrets.ymlfile.Restart all airflow containers using:phi ws restart --name airflow --type containerTest a DAG# ssh into airflow-worker | airflow-wsdockerexec-itairflow-ws-containerzsh
dockerexec-itairflow-worker-containerzsh# Test run the DAGs using module namepython-mworkflow.dir.file# Test run the DAG filepython/mnt/workspaces/data-platform/workflow/dir/file.py# List DAGsairflowdagslist# List tasks in DAGairflowtaskslist\-S/mnt/workspaces/data-platform/workflow/dir/file.py\-tdag_name# Test airflow taskairflowtaskstestdag_nametask_name2022-07-01Recreate everythingNotes:Usedata-platformas the working directoryDeactivate existing venv usingdeactivateif neededecho"*- Deleting venv"rm-rf~/.venvs/dpenvecho"*- Deleting af-db-dp-volume volume"dockervolumermaf-db-dp-volumeecho"*- Recreating venv"python3-mvenv~/.venvs/dpenvsource~/.venvs/dpenv/bin/activateecho"*- Install phi"pipinstallphidata
phiinitecho"*- Setup + deploying workspace"phiwssetupCACHE=fphiwsup