Package: atd-knack-services
atd-knack-servicesIntegration services for ATD's Knack applications.DesignATD Knack Services is comprised of a Python library (/services) and scripts (/scripts) which automate the flow of data from ATD's Knack applications to downstream systems.These utilities are designed to:incrementally offload Knack application records and metadata as a JSON documents in a collection of S3 data storesincrementally fetch records and publish them to external systems such as Socrata and ArcGIS Onlinelay the groundwork for further integration with a data lake and/or a data warehousebe deployed in Airflow or similar task management frameworksConfigurationS3 Data StoreData is stored in an S3 bucket (s3://atd-knack-services), with one subdirectory per Knack application per environment. Each app subdirectory contains a subdirectory for each container, which holds invdividual records stored as JSON a file with itsidserving as the filename. As such, each store follows the naming patterns3://atd-knack-servies/<app-name>-<environment>/<container ID>.Application metadata is also stored as a JSON file at the root of each S3 bucket.. s3://atd-knack-services
|- data-tracker-prod
|   |-- 2x22pl1f7a63815efqx33p90.json   #  app metadata
|   |-- view_1
|       |-- 5f31673f7a63820015ef4c85.json
|       |-- 5b34fbc85295dx37f1402543.json
|       |-- 5b34fbc85295de37y1402337.json
|       |...Scripts (/scripts)Get the most recent successful DAG runmost_recent_dag_run.pyis meant to be run as an initial Airflow task which fetches the most recent successful run of itself. The date can then be passed to subsequent tasks as a filter parameter to support incremental record processing.$pythonmost_recent_dag_run.py--dagatd_signals_socrataCLI arguments--dag(str, required): the DAG ID of DAG run to be fetched.Load App Metadata to S3Useupload_metadata.pyto load an application's metadata to S3.$pythonupload_metaddata.py\--app-namedata-tracker\--envprod\CLI arguments--app-name(str, required): the name of the source Knack application--env(str, required): The application environment. Must beprodordev.Load Knack Records to S3Useknack_container_to_s3.pyto incrementally load data from a Knack container (an object or view) to an S3 bucket.$pythonknack_container_to_s3.py\--app-namedata-tracker\--containerview_197\--envprod\--date1598387119\Publish Records to the Open Data PortalUseupsert_knack_container_to_socrata.pyto publish a Knack container to the Open Data Portal (aka, Socrata).$pythonupsert_knack_container_to_socrata.py\--app-namedata-tracker\--containerview_197\--envprod\--date1598387119\CLI arguments--app-name(str, required): the name of the source Knack application--container(str, required): the name of the object or view key of the source container--env(str, required): The application environment. Must beprodordev.--date(int, required): a POSIX timestamp. only records which were modified at or after this date will be processed.Services (/services)The services package contains utilities for fetching and pushing data between Knack applications and AWS S3.It is designed as a free-standing Python package can be installed withpip:$pipinstallatd-knack-servicesand imported asservices:importservicesservices.s3.uploadMulti-threaded uploading of file-like objects to S3.services.s3.downloadMulti-threaded downloading of file objects from S3.How ToCreate bucket(s)Add Knack app credentials to auth configuration fileAdd container configuration file to /services/configCreate DAGsAn end-to-end ETL process will involve creating at least three Airflow tasks:Load app metadata to S3Load Knack records to S3Publish Knack records to destination system