Package: astro-extras
Astro SDK extras projectWe've been usingApache Airflowas ETL tool for quite some time. Recently I came acrossAstronomer's Astro SDKwhich simplifies data processig with Airflow very much by providing
functionality to easily load or save data,
define custom transformations, adding transparent support of newestDatasetconcept, automatic lineage and so on.The goal of this project is to add some usefull features to these great products, such as:ETL session conceptSQL file-based templates supportSimplified table processing routinesHeterogenios data transferand morePlease note that this project is in the very early stage of development
so anything could change in the future.ConceptsETL SessionETL session is a logical group of data transfers united by single identifier (session_id).
Sessions store information about data transfer source, target, data loading period
and completion state thus providing all necessary information about the data flow.ETL session object defines its own data loading interval and allows
to override automatic interval boundaries calculation using parameters when a DAG is started.Sessions are stored in a database insessionstable. Entries to that table are created
when a new session is instantiated withing a DAG run and closed when DAG
is finished.ETL session instance is pushed to XCom undersessionkey
and becomes available to all operators within the data pipeline (see query example below).SQL TemplatesSQL template is a file, which contains instructions and Jinja-macros substituted at runtime.
For example, this template could be created in order to load data fromdata_tabletable
within the limited date range:select*fromdata_tablewheresome_datebetween'{{ ti.xcom_pull(key="session").period_start }}'::timestampand'{{ ti.xcom_pull(key="session").period_end }}'::timestampIn order to be used, template file must be named after target object name
and placed in atemplates\<dag_id>folder under DAGS_ROOT. In given case,
assuming it will be used intest-data-loadDAG,
template file name must be./dags/templates/data_table.sql.Note that it uses anETLSessionobject from XCom to filter only actual data.Heterogenuos data transferOriginal Astro-SDK misses functionality to transfer data between different
database systems (for example, from Oracle to PostgreSQL or vise versa).However, there is aGenericTransferoperator available in original Airflow code, so this package adopts it to
Astro-SDK style providing some easy-to-usertransfer_tableandtransfer_tablesfunctions.
They are highly customizable and fully support SQL templates.Timed dictionaries supportTimed dictionaries are special kind of tables used mostly in Data Warehousing.
They are designed in such a way that they keep multiple values of the same
attribute for given key, and have a timestamp on what attribute value was
at particular time moment.For example, this is a time dictionary table (PostgreSQL notation is used):createtabledict_values(record_idserialprimarykey,effective_fromtimestampnotnull,effective_totimestamp,src_idintnotnull,some_valuetextnotnull);Here,src_idis an ID of a record on a source, andsome_valueis
time-dependent attribute of that ID.
And,effective_from/effective_topair of columns denotes
what period the particular value ofsome_valuewas assigned to
particularsrc_idvalue.If we have this table filled like this:record_ideffective_fromeffective_tosrc_idsome_value12023-01-11 00:00:00null10Some valueand thensome_valueproperty of record withid = 10on source changes to `Some other value``,
then, after updating at 2023-11-02 00:01:00, this timed dictionary should look like this:record_ideffective_fromeffective_tosrc_idsome_value12023-11-01 00:00:012023-11-02 00:00:5910Some value22023-11-02 00:01:00null10Some other valueAs you see, record withrecord_id = 1is now "closed", meaning that it
haseffective_toattribute set, and new record withrecord_id = 2was
added and haveeffective_toset to null.Any query on this timed dictionarty should includeeffective_from/effective_tocolumns check, for example like this:selectsrc_idasid,some_valuefromdict_valueswheredate_of_interestbetweeneffective_fromandcoalesce(effective_to,'2099-12-31')where date_of_interest is some date when this query is running for.To update timed dictionaries, one should useupdate_timed_table/update_timed_tablesfunctions.Usage examplesCreate a DAG which opens a session, outputs session info to log and closes it:withDAG(...)asdag,ETLSession('source','target')assession:@dag.taskdefprint_session(session:ETLSession):print(session)print_session(session)Create a DAG withopen-session -> transfer-data_table -> close_sessiontask sequence:withDAG(...)asdag:session=open_session('source','target')transfer_table('data_table',session=session)close_session(session)Upon execution, this DAG will transfer data ofdata_tablefrom source to target
database wrapping this in a session. This would help to easily identify when particular record
was loaded or clean up after unsuccessfull attempts.The same, but using a context manager:withDAG(...)asdag,ETLSession('source','target')assession:transfer_table('test_table',session=session)InstallationTo build a package from source, run Python build command and then install the package:python -m build .pip install ./dist/astro_extras-xxxx-py3-none-any.whlAirflow could be used directly under Unix environment, but this is not possible in Windows.
Anyway, using a docker is much easier in all cases.Adocker-compose.ymlfile should be used to start Airflow in a docker
environment. After building the package, run this command at the top directory:docker compose -p astro-extras up -d --buildTo shutdown the docker, run:docker compose -p astro-extras down -vTestingTo setup testing evironment, install pytest with:pip install -r ./tests/requirements.txtThen, jump to/testsdirectory and run pytest:cd ./testspytest -s -vThis will build docker image and startastro-extras-teststack usingdocker-compose-test.ymlfile.
The packaget must be built beforehand.After the tests complete, test stack will be shot down, but you may add--keepoption
in order to keep it running.