Package: dynamicio
A repository for hosting thedynamiciolibrary, used as a wrapper forpandasi/o operations.-- Logo illustrated byNick LoucasTable of ContentsWhy wrap your i/o?Managing various ResourcesManaging Various Data TypesValidations & Metrics GenerationTesting (Running Local Regression Tests)So, what do we do about these?The SolutionMain featuresSupported sources and data formats:Coming soonInstallationAPI DocumentationHow to useKeywordsLet's startStep 1: Resource DefinitionsStep 2: Defining your environment variablesStep 3: Read in your resource definitionsStep 4: Loading the data resourcesStep 4.1.SCHEMA_FROM_FILEStep 4.2. Use the dynamicio cliStep 4.3: Loading fromS3Step 4.3: Loading fromPostgresStep 5: Writing outStep 6: Full CodeUtilisingasyncioTesting LocallyLast notesTable of contents generated with markdown-tocWhy wrap your i/o?Working withpandasdataframes has opened up a new world of potential in Data Science. However,
if you are usingpandasto support production pipelines, whether ML or ETL, you end up having
a big part of your code be concerned with I/O operations.Managing various ResourcesFirst, it's the various type of resources you need to interact with; object storage (S3 or GCS)
databases (Athena, Big Query, Postgres), Kafka and many more. For each of these, you have
dependencies on various libraries such ass3fs,fsspec,gcfs,boto3,awscli,aws-wrangler,sql-alchemy,tables,kafka-pythonand many more.Managing Various Data TypesThen it's the various data types you need handle,parquet,hdfs,csv,jsonand many
others, each of which come with their own set of configurationkwargs, things like the
orientation of the dataset (json) or the parquet engine you want
to use behind the scenes (pyarroworfastparquet).Validations & Metrics GenerationThen, it's the need to validate your expectations on the datasets; things like unique or
null values being allowed in a column, allowing only a specific set of categorical values, or
numerical values within a specified range. And what about metrics generation? The ability to
monitor data distributions and how various metrics change with every run, is a significant
aspect of monitoring the quality of your solution.Testing (Running Local Regression Tests)Finally, what about testing your code in different environments? Take, for instance, a traditional
setup where you have the following 4 environments to work against:Local;Develop;Staging, and;Production.Configuring your code to work against the last 3Develop, Staging and Productioncan easily be done through
environment variables, but what about testing locally? What if you want to run your pipelines locally?
Well, you can, but usually that entails a big deal of mocking calls to external services. Instead, wouldn't
it be great if you could seamlessly direct your I/O operations to local sample data.So, what do we do about these?This proliferation of I/O operations leads to the emergence of glue code, which can be very difficult to manage. The problem is highlighted
as anML-System Anti-PatterninHidden Technical Debt in Machine Learning Systems...ML researchers tend to develop general purpose solutions as self-contained packages. A wide variety of these are
available as open-source packages at places likeml-oss.org, or from in-house code, proprietary packages, and cloud-based platforms.Using generic packages often results in aglue codesystem design pattern, in which a massive amount of supporting
code is written to get data into and out of general-purpose packages.Glue codeis costly in the long
term because it tends to freeze a system to the peculiarities of a specific package; testing alternatives
may become prohibitively expensive. In this way, using a generic package can inhibit improvements, because it
makes it harder to take advantage of domain-specific properties or to tweak the objective function to
achieve a domain-specific goal. Because a mature system might end up being (at most) 5% machine learning
code and (at least) 95% glue code, it may be less costly to create a clean native solution rather
than re-use a generic package.The SolutionQuoting from the same paper:An important strategy for combating glue-code is to wrap black-box packages into common API's. This allows supporting
infrastructure to be more reusable and reduces the cost of changing packages.Dynamicio (or dynamic(i/o)) serves exactly that; it serves as a convenient wrapper aroundpandasI/O operations. It's a manifestation of
the dependency inversion principle--a layer of indirection if you want--which keeps your code DRY and increases re-usability, effectively
decoupling business logic from the I/O layer.Main featuresdynamic(i/o)supports:seamless transition between environments;abstracting away from resource and data types throughresource definitions;honouring your expectations on data throughschema definitions;metrics auto-generation (logging) for monitoring purposes.Supported sources and data formatsS3(or local) Input & Output:parqueth5jsoncsvPostgresInput & OutputKafkaOutputComing soonAthena(pending)Delta Tables(pending)GCS(pending)BigQuery(pending)InstallationTo installdynamic(i/o)you need to first authenticate with AWS Code Artifact. Just follow the below steps:>>pipinstalldynamicioAPI DocumentationRead our docs here:https://vortechsa.github.io/dynamicio/How to useWe will go over an end-to-end example for reading and writing a single dataset, covering:all components involved and how they are configured, and;how these components are pieced togetherYou can find this example under the demo directory fo this repo.Keywords:source configsresource definitionsschema definitionsLet's startSuppose you want to ingest thefooandbardatasets, respectively fromS3andPostgresand stage them
to S3 for further processing.Assume you want to build a pipeline that looks something like the image below:Assume the below repository structure, which implements this pipeline, for the purpose of this tutorial:demo
├──__init__.py
├──resources
│├──definitions
││├──input.yaml
││├──processed.yaml
││└──raw.yaml
│└──schemas
│├──input
││├──bar.yaml
││└──foo.yaml
│└──processed
│├──final_bar.yaml
│└──final_foo.yaml
├──src
│├──__init__.py
│├──__main__.py
│├──constants.py
│├──environment.py
│├──io.py
│├──runner_selection.py
│└──runners
│├──__init__.py
│├──staging.py
│└──transform.py
└──tests├──__init__.py├──conftest.py├──constants.py├──data│├──input││├──bar.parquet││└──foo.csv│├──processed││└──expected││├──final_bar.parquet││└──final_foo.parquet│└──raw│└──expected│├──staged_bar.parquet│└──staged_foo.parquet├──runners│├──__init__.py│├──conftest.py│├──test_staging.py│└──test_transform.py├──test_pipeline.py└──test_runner_selection.pyStep 1: Resource DefinitionsWe will start with defining our input and output resources as yaml files. These need to be defined underresources/definitions:resources
├──__init__.py
├──definitions
│├──input.yaml
│├──processed.yaml
│└──raw.yaml
└──schemas├──input│├──bar.yaml│└──foo.yaml└──processed├──final_bar.yaml└──final_foo.yamlYou will need to define your pipeline's resources by creating threeyamlfiles. The first is:input.yamlwhich concerns data read by thestagingtask;---FOO:sample:type:"local"local:file_path:"[[TEST_RESOURCES]]/data/input/foo.csv"file_type:"csv"actual:type:"s3"s3:bucket:"[[S3_YOUR_INPUT_BUCKET]]"file_path:"data/foo.h5"file_type:"hdf"schema:file_path:"[[RESOURCES]]/schemas/input/foo.yaml"BAR:sample:type:"local"local:file_path:"[[TEST_RESOURCES]]/data/input/bar.parquet"file_type:"parquet"actual:type:"postgres"postgres:db_host:"[[DB_HOST]]"db_port:"[[DB_PORT]]"db_name:"[[DB_NAME]]"db_user:"[[DB_USER]]"db_password:"[[DB_PASS]]"schema:file_path:"[[RESOURCES]]/schemas/input/bar.yaml"theraw.yaml, which concerns data coming out of thestagingtask and go into thetransformtask:---STAGED_FOO:sample:type:"local"local:file_path:"[[TEST_RESOURCES]]/data/raw/staged_foo.parquet"file_type:"parquet"actual:type:"s3"s3:bucket:"[[S3_YOUR_OUTPUT_BUCKET]]"file_path:"live/data/raw/staged_foo.parquet"file_type:"parquet"STAGED_BAR:sample:type:"local"local:file_path:"[[TEST_RESOURCES]]/data/raw/staged_bar.parquet"file_type:"parquet"actual:type:"s3"s3:bucket:"[[S3_YOUR_OUTPUT_BUCKET]]"file_path:"live/data/raw/staged_bar.parquet"file_type:"parquet"and theprocessed.yaml, which concerns data coming out of the *transformtask:---FINAL_FOO:sample:type:"local"local:file_path:"[[TEST_RESOURCES]]/data/processed/final_foo.parquet"file_type:"parquet"actual:type:"s3"s3:bucket:"[[S3_YOUR_OUTPUT_BUCKET]]"file_path:"live/data/processed/final_foo.parquet"file_type:"parquet"schema:file_path:"[[RESOURCES]]/schemas/processed/final_foo.yaml"FINAL_BAR:sample:type:"local"local:file_path:"[[TEST_RESOURCES]]/data/processed/final_bar.parquet"file_type:"parquet"options:use_deprecated_int96_timestamps:truecoerce_timestamps:"ms"allow_truncated_timestamps:falserow_group_size:1000000actual:type:"kafka"kafka:kafka_server:"[[KAFKA_SERVER]]"kafka_topic:"[[KAFKA_TOPIC]]"options:compression_type:"snappy"max_in_flight_requests_per_connection:10batch_size:262144request_timeout_ms:60000# 60sbuffer_memory:134217728# 128MBschema:file_path:"[[RESOURCES]]/schemas/processed/final_bar.yaml"We will hence refer to these files as"resource definitions".The first,input.yamldefines the input sources for thestagingtask, handled by the respective module (runners/staging.py) while the
second one, defines its output; similarly fortransform. These files are parsed bydynamicio.config.IOConfigto generated configuration i/o
instances referred to as "source configs" (seedemo/src/__init__.py).Notice that under every source there are three layers:sample,actualandschema. The first two point to the variants of the same
dataset, depending on whether it is called from the local environment or from the cloud (we will showcase how this
distinction takes place later).The third refers your source config to a"schema definition"for your dataset (we will cover this in detail later).Step 2: Defining your environment variablesAlso notice that paths to datasets are embedded with dynamic values identified with double squared brackets, e.g.[[ S3_YOUR_OUTPUT_BUCKET ]]. These can be defined in a module in your repository.Resource definitions (*.yamlfiles) work in conjunction withglobalandenvironmentvariables:environment.py├──__init__.py
├──src└──environment.py
...Let's have a look inside."""A module for configuring all environment variables."""importosENVIRONMENT="sample"CLOUD_ENV="DEV"RESOURCES=os.path.join(os.path.dirname(os.path.realpath(__file__)),"../resources")TEST_RESOURCES=os.path.join(os.path.dirname(os.path.realpath(__file__)),"../tests")S3_YOUR_INPUT_BUCKET=NoneS3_YOUR_OUTPUT_BUCKET=NoneKAFKA_SERVER=NoneKAFKA_TOPIC=NoneDB_HOST=NoneDB_PORT=NoneDB_NAME=NoneDB_USER=NoneDB_PASS=NoneREFERENCE_DATA_STATE_KEY=NoneLOWER_THAN_LIMIT=1000# We will discuss this one later in step 4.This module will be passed as an input parameter to instances of thedynamicio.config.IOConfigclass. Let's cover
some of its variables:"ENVIRONMENT":"sample",used to distinguish between local and cloud runs of your module. It assumes that this environment variable is
defined in the cloud environment where your module is executed from."TEST_RESOURCES":os.path.join(os.path.dirname(os.path.realpath(__file__)),"../tests"),It is defined in the resource definitions, e.g.:---FOO:sample:type:"local"local:file_path:"[[TEST_RESOURCES]]/data/input/foo.csv"file_type:"csv"actual:type:"s3"s3:bucket:"[[S3_YOUR_INPUT_BUCKET]]"file_path:"data/foo.h5"file_type:"hdf"schema:file_path:"[[RESOURCES]]/schemas/input/foo.yaml"and therefore needs to be defined here as well.Any other dynamic variable (identified with the doubly squared brackets) defined in the resource definitions needs to
also be defined here and can be either statically or dynamically defined (i.e. hardcoded or defined as an environment
value).Step 3: Read in your resource definitionsReading in the resources definitions can be done by means of instantiating instances of thedynamicio.config.IOConfigclass (the, so called, "source configs"). This is done in:src
├──__init__.pywhich allows it to be automatically loaded on call of any module within thepipelinepackage."""Set config IOs"""__all__=["input_config","raw_config","processed_config"]importloggingimportosfromdemo.srcimportenvironmentfromdemo.src.environmentimportENVIRONMENT,RESOURCESfromdynamicio.configimportIOConfiglogging.basicConfig(level=logging.INFO)logging.getLogger("kafka").setLevel(logging.WARNING)input_config=IOConfig(path_to_source_yaml=(os.path.join(RESOURCES,"definitions/input.yaml")),env_identifier=ENVIRONMENT,dynamic_vars=environment,)raw_config=IOConfig(path_to_source_yaml=(os.path.join(RESOURCES,"definitions/raw.yaml")),env_identifier=ENVIRONMENT,dynamic_vars=environment,)processed_config=IOConfig(path_to_source_yaml=(os.path.join(RESOURCES,"definitions/processed.yaml")),env_identifier=ENVIRONMENT,dynamic_vars=environment,)On loading,IOConfigwill load the respective configs for all resources in the form of a multi-layered dictionary, e.g.,
foractual:importdemo.src.environment{"FOO":{"sample":{"type":"local","local":{"file_path":f"{demo.src.environment.TEST_RESOURCES}/data/input/foo.csv","file_type":"csv",},},"actual":{"type":"s3","s3":{"bucket":f"{demo.src.environment.S3_YOUR_INPUT_BUCKET}","file_path":"data/foo.h5","file_type":"hdf"}},}}Then, depending on the value of theenv_identifierparameter, the respective sub-dictionary is returned. For example,
with:foo_io=input_config.get(source_key="FOO")and withenv_identifier="actual", the output would be:"type":"s3","s3":{"bucket":f"{demo.src.environment.S3_YOUR_INPUT_BUCKET}","file_path":"data/foo.h5","file_type":"hdf"}Step 4: Loading the data resourcesTo load a resource, you will need to generate instances of subclasses offrom dynamicio import UnifiedIOclass. Note
that theUnifiedIOclass operates as an abstract class and cannot be used for instantiating objects.You will need to implement your own subclasses for each of the inputs you care to load. You can do this in theio.pymodule, under:.
├──src
│├──__init__.py
│├──io.pyThe file looks like this:"""Responsible for configuring io operations for input data."""# pylint: disable=too-few-public-methods__all__=["InputIO","StagedFoo","StagedBar"]fromsqlalchemy.ext.declarativeimportdeclarative_basefromdynamicioimportUnifiedIO,WithLocal,WithPostgres,WithS3Filefromdynamicio.coreimportSCHEMA_FROM_FILE,DynamicDataIOBase=declarative_base()classInputIO(UnifiedIO):"""UnifiedIO subclass for V6 data."""schema=SCHEMA_FROM_FILEclassStagedFoo(WithS3File,WithLocal,DynamicDataIO):"""UnifiedIO subclass for staged foos."""schema={"column_a":"object","column_b":"object","column_c":"int64","column_d":"int64",}classStagedBar(WithLocal,WithPostgres,DynamicDataIO):"""UnifiedIO subclass for cargo movements volumes data."""schema={"column_a":"object","column_b":"object","column_c":"int64","column_d":"int64",}Instances of theDynamicDataIOclass can either inherit directly fromUnifiedIO(e.g.InputIOinherits fromUnifiedIO) or user can choose the mixins they want to use
(e.g.StagedFooinherits fromWithS3FileandWithLocalmixins and needs to inherit fromDynamicDataIO; note that MOR kicks in to address polymorphic conflicts--i.e. order matters).
Also, all instances of theDynamicDataIOmustdefine a classschema. The schema can have the form of a dictionary, associating columns (keys) withdtypes(values) or be defined
as a yaml file (seeInputIO) as explained in the next section.N.B.For convenience's sake and to reduce the need of boilerplate code, using a single class definition likeInputIOis recommended (this way all your datasets can be loaded with instances of the same class).
However, if you need to use different mixins for different datasets, you can do so by defining a class for each dataset (e.g.StagedFooandStagedBar). You will definitely need to define your own classes if you
want to avoid usingSCHEMA_FROM_FILE(as per the below instructions) but in this case, your dataset's name will be inferred from the dataclass name you use, e.g.StagedFoowill be inferred asSTAGED_FOO.Step 4.1.SCHEMA_FROM_FILEfrom dynamicio.core import SCHEMA_FROM_FILEis a unique dynamic(i/o) object used as a placeholder.
It is used to indicate that a schema is provided as part of aresource definition.For example:---FOO:sample:...actual:...schema:file_path:"[[RESOURCES]]/schemas/input/foo.yaml"foo.yamlis effectively a schema definition and looks like this:---name:foocolumns:column_a:type:"object"validations:has_unique_values:apply:trueoptions:{}metrics:-Countscolumn_b:type:"object"validations:has_no_null_values:apply:trueoptions:{}metrics:-CountsPerLabelcolumn_c:type:float64validations:is_greater_than:apply:trueoptions:threshold:1000metrics:[]column_d:type:float64validations:is_lower_than:apply:trueoptions:threshold:1000metrics:-Min-Max-Mean-Std-VarianceThe file is quite self-explanatory. The format is:DataSet:ColumntypevalidationsmetricsFor a dataset, each of the desired columns are dictated here, along with their
designateddtypes. Thecolumnsare used to filter out undesired columns in an optimal manner.
This means that it will happen on loading for*.csvand*.parquetfiles as well as when
interacting with a database, but will happen post-loading in the case of*.h5or*.json.dtypesare then used to validate the types of the columns. If types don't match,dynamic(i/o)will attempt to cast them and will issue aWARNING. If casting does not work either, it will
throw aValueErrorexception.validationsandmetricsare there to document the user's expectations of the quality of the dataset.
They can be automatically applied on loading or on writing out.Specifically, you can use the followingvalidations:has_unique_values# no optionshas_no_null_values# no optionshas_acceptable_percentage_of_nullsis_in:validations:is_in:apply:trueoptions:categorical_values:-class_a-class_b-class_cmatch_all:false# true by default, if false, then the column unique categoricals must be equal to the acceptable ones, else they must be a subsetis_greater_thanvalidations:is_greater_than:apply:trueoptions:threshold:1000is_greater_than_or_equal# same asis_greater_thanis_lower_than# same asis_greater_thanis_lower_than_or_equal# same asis_greater_thanis_between# same asis_greater_thanvalidations:is_between:apply:trueoptions:lower:0upper:1000include_left:falseinclude_right:true# true by defaultandmetrics:MinMaxMeanStdVarianceCountsUniqueCountsCountsPerLabelimposed as per below:column_c:type:float64validations:{}metrics:-Min-Max-Mean-Std-...Note that you can also use dynamic fields to define validations, e.g. seeLOWER_THAN_LIMITin the file below:---name:barcolumns:column_a:type:"object"validations:has_unique_values:apply:trueoptions:{}metrics:-Countscolumn_b:type:"object"validations:has_no_null_values:apply:trueoptions:{}metrics:-CountsPerLabelcolumn_c:type:float64validations:is_greater_than:apply:trueoptions:threshold:1000metrics:[]column_d:type:float64validations:is_lower_than:apply:trueoptions:threshold:"[[LOWER_THAN_LIMIT]]"metrics:-Min-Max-Mean-Std-VarianceSimilar to resource definitions, this value needs to be defined inenvironment.pyStep 4.2. Use the dynamicio cliThedynamiciocli can be used to automatically generate schema definitions for you, provided either a path to
a dataset (json,parquet,hdf,csv) or to a directory. Here is how you can use it:usage:dynamicio[-h](--batch|--single)-pPATH-oOUTPUT

Generatedatasetschemas

optionalarguments:-h,--helpshowthishelpmessageandexit--batchusedtogeneratemultipleschemasprovidedadatasetsdirectory.--singleusedtogenerateaschemaprovidedasingledataset.-pPATH,--pathPATHthepathtothedataset/datasets-directory.-oOUTPUT,--outputOUTPUTthepathtotheschemasoutputdirectory.The generated schema definitions will not have any validations or metrics automatically selected for you.Step 4.3: Loading fromS3To then load fromS3you simply do:foo_df=InputIO(source_config=input_config.get(source_key="FOO"),apply_schema_validations=True,log_schema_metrics=True).read()which will load thefoo.csvfile as a dataframe.Step 4.3: Loading fromPostgresLikewise toS3resources,postgresresources need the same number of options to be defined for their loading.Implicitly, dynamicio is able to infer data model from the schema yml files of the source key provided rather than requiring that the schema is explicitly defined.This data model defines the table, the columns and their respective SQL types.To, then, load frompostgresyou simply do:bar_df=InputIOsource_config=input_config.get(source_key="BAR"),apply_schema_validations=True,log_schema_metrics=True).read()which will load the cargo the movements table as a dataframe.Step 5: Writing outSinking data is done in a very similar way. You need to:Define your output resource definitions, in our case inraw.yaml---STAGED_FOO:sample:type:"local"local:file_path:"[[TEST_RESOURCES]]/data/raw/staged_foo.parquet"file_type:"parquet"actual:type:"s3"s3:bucket:"[[S3_YOUR_OUTPUT_BUCKET]]"file_path:"live/data/raw/staged_foo.parquet"file_type:"parquet"STAGED_BAR:sample:type:"local"local:file_path:"[[TEST_RESOURCES]]/data/raw/staged_bar.parquet"file_type:"parquet"actual:type:"s3"s3:bucket:"[[S3_YOUR_OUTPUT_BUCKET]]"file_path:"live/data/raw/staged_bar.parquet"file_type:"parquet"You need to define the respective dynamic values found in your resource definitions
in yoursrc/environment.pyYou need to create an instance of theIOConfigclass for theraw.yamlin the__init__.pyfile (we already did this).Define the additionalDynamicDataIOsubclasses in thesrc/io.pymodule,
dictating through the schema the list of columns, and their types (also used for schema validation).Finally, instantiate instances of those subclasses and call the.write()method,
passing in the dataframe you want to write out, e.g.demo/src/runners/staging.py:...StagedFoo(source_config=raw_config.get(source_key="STAGED_FOO"),**constants.TO_PARQUET_KWARGS).write(foo_df)StagedBar(source_config=raw_config.get(source_key="STAGED_BAR")).write(bar_df)Notice that you can pass allpandasoptions to write out, when for instance you are writing outparquet.demo/src/constants.py:# ParquetTO_PARQUET_KWARGS={"use_deprecated_int96_timestamps":False,"coerce_timestamps":"ms","allow_truncated_timestamps":True,}Of, course this is not a problem as parquet is the format used by both resources in either environment. This not always the case however. See indemo/resources/definitions/processed.yaml:---...FINAL_BAR:sample:type:"local"local:file_path:"[[TEST_RESOURCES]]/data/processed/final_bar.parquet"file_type:"parquet"options:<---- Options for Local writing as parquetuse_deprecated_int96_timestamps:truecoerce_timestamps:"ms"allow_truncated_timestamps:falserow_group_size:1000000actual:type:"kafka"kafka:kafka_server:"[[KAFKA_SERVER]]"kafka_topic:"[[KAFKA_TOPIC]]"options:<---- Options for writting to a Kafka Topiccompression_type:"snappy"max_in_flight_requests_per_connection:10batch_size:262144request_timeout_ms:60000# 60sbuffer_memory:134217728# 128MBschema:file_path:"[[RESOURCES]]/schemas/processed/final_bar.yaml"Here, we have a case where different options need to be used for each environment as it deals with a different source. This is gracefully managed through resource
definitions passing these arguments in theoptionskey per environment.Step 6: Full CodeThe full code for the loading module in our example would live under:├──__init__.py
├──src
...
│├──runners
││└──staging.pyand looks like:"""Add module docstring...."""importloggingfromdemo.srcimportconstants,input_config,raw_configfromdemo.src.ioimportInputIO,StagedBar,StagedFoologger=logging.getLogger(__name__)defmain()->None:"""The entry point for the Airflow Staging task.Returns:Void function."""# LOAD DATAlogger.info("Loading data from live sources...")bar_df=InputIO(source_config=input_config.get(source_key="BAR"),apply_schema_validations=True,log_schema_metrics=True).read()foo_df=InputIO(source_config=input_config.get(source_key="FOO"),apply_schema_validations=True,log_schema_metrics=True).read()logger.info("Data successfully loaded from live sources...")# TRANSFORM  DATAlogger.info("Apply transformations...")# TODO: Apply your transformationslogger.info("Transformations applied successfully...")# SINK DATAlogger.info("Begin sinking data to staging area:")StagedFoo(source_config=raw_config.get(source_key="STAGED_FOO"),**constants.TO_PARQUET_KWARGS).write(foo_df)StagedBar(source_config=raw_config.get(source_key="STAGED_BAR")).write(bar_df)logger.info("Data staging is complete...")UtilisingasyncioDynamic(i/o)supports use ofasyncioto speed upI/O boundoperations through leveraging multithreading.An example can be found in the second of the two demo tasks, namely, thetransform.pytask."""Add module docstring...."""importasyncioimportloggingimportdemo.src.environmentfromdemo.srcimportprocessed_config,raw_configfromdemo.src.ioimportInputIO,StagedBar,StagedFoologger=logging.getLogger(__name__)asyncdefmain()->None:"""The entry point for the Airflow Staging task.Returns:Void function."""# LOAD DATAlogger.info("Loading data from live sources...")[bar_df,foo_df]=awaitasyncio.gather(StagedBar(source_config=raw_config.get(source_key="STAGED_BAR")).async_read(),StagedFoo(source_config=raw_config.get(source_key="STAGED_FOO")).async_read())logger.info("Data successfully loaded from live sources...")# TRANSFORM  DATAlogger.info("Apply transformations...")# TODO: Apply your transformationslogger.info("Transformations applied successfully...")# SINK DATAlogger.info(f"Begin sinking data to staging area: S3:{demo.src.environment.S3_YOUR_OUTPUT_BUCKET}:live/data/raw")awaitasyncio.gather(InputIO(source_config=processed_config.get(source_key="FINAL_FOO"),apply_schema_validations=True,log_schema_metrics=True).async_write(foo_df),InputIO(source_config=processed_config.get(source_key="FINAL_BAR"),apply_schema_validations=True,log_schema_metrics=True).async_write(bar_df),)logger.info("Data staging is complete...")In short, you simply need to utilise theasync_read()or theasync_write()methods instead, plus await and gather your calls.Testing LocallyAfter following the above documentation, at this point it should be clear thatdynamic(i/o)is optimised for enabling
seamless local testing for your pipelines.Simply by configuring yourENVIRONMENT's default value tosampleand provided that you have the required tests data
sources in the necessary directories, it becomes very simple to test your pipelines end-to-end in seconds, eliminating
the need to deploy your dags and wait for their tasks to be provided access to processing resources.All you need to do is mimic the order of execution of your tasks, running them in procedural order.In the case of our example, you would have to:Add the necessary data undertests/data:└──tests├──__init__.py├──conftest.py├──constants.py├──data│├──input││├──bar.parquet││└──foo.csv│├──processed││└──expected││├──final_bar.parquet││└──final_foo.parquet│└──raw│└──expected│├──staged_bar.parquet│└──staged_foo.parquet├──runners│├──__init__.py│├──conftest.py│├──test_staging.py│└──test_transform.py├──test_pipeline.py└──test_runner_selection.pyImplement an end-to-end, black-box style test that simply generates the expected data output given a
specific input (deleting the output after the assertion)An example end-to-end test in this case, for a single airflow task would look like:"""An example pipeline to showcase how dynamicio can bt used for setting up a local e2e testing!"""# pylint: disable=missing-module-docstring, missing-class-docstring, missing-function-docstring, unused-argument, too-few-public-methods# noqaimportosimportpandasaspdimportpytestfromdemo.srcimportprocessed_config,raw_configfromdemo.src.runnersimportstaging,transformclassTestPipeline:"""Example e2e test."""@pytest.mark.end_to_enddeftest_dag_with_mock_sample_input_data(self,expected_staged_foo_df,expected_staged_bar_df,expected_final_foo_df,expected_final_bar_df,):"""Showcases how you can leverage dynamicio to read local data for fast feedback when you want to run your pipelines locally."""# Given# The src/resources/input.yaml# Whenstaging.main()transform.main()# Thentry:assertexpected_staged_foo_df.equals(pd.read_parquet(raw_config.get(source_key="STAGED_FOO")["local"]["file_path"]))assertexpected_staged_bar_df.equals(pd.read_parquet(raw_config.get(source_key="STAGED_BAR")["local"]["file_path"]))assertexpected_final_foo_df.equals(pd.read_parquet(processed_config.get(source_key="FINAL_FOO")["local"]["file_path"]))assertexpected_final_bar_df.equals(pd.read_parquet(processed_config.get(source_key="FINAL_BAR")["local"]["file_path"]))finally:os.remove(raw_config.get(source_key="STAGED_FOO")["local"]["file_path"])os.remove(raw_config.get(source_key="STAGED_BAR")["local"]["file_path"])os.remove(processed_config.get(source_key="FINAL_FOO")["local"]["file_path"])os.remove(processed_config.get(source_key="FINAL_BAR")["local"]["file_path"])Last notesHope this was helpful.Please do reach out with comments and your views about how the library or the docs can be improved, and by all means, come along and contribute to our project!